[Geoserver-users] Rasters, Tiles, and FME

[Jonathan Moules]

Hi List,
A few generic questions.

First: What are gridsets actually for? I know how to create them (the
documentation is very good for that), but I can't figure out what they're
for.

Second: I have a number of large rasters in JPEG2000 format (up to several
GB in size). GeoServer can't access these (we're not going to use the
extension). I have FME so can do lots of processing on them if needed:
What's the best way to serve these up? I was thinking of tiling them (WMS
can serve tiled data right?). But then the question becomes, which tiling
system do I use? There seem to be several:

imagePyramid -
http://docs.geoserver.org/stable/en/user/tutorials/imagepyramid/imagepyramid.html?highlight=tiling
JDBC -
http://docs.geoserver.org/stable/en/user/tutorials/imagemosaic-jdbc/imagemosaic-jdbc_tutorial.html?highlight=tiling
imageMosaic -
http://docs.geoserver.org/stable/en/user/tutorials/image_mosaic_plugin/imagemosaic.html

What are the advantages/disadvantages of each? I'm guessing some are easier
to create/use and possibly faster.

Thanks,
Jonathan


This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

[Arne Kepp]

If you want to have efficient caching it is crucial that clients ask for
the tiles that you have already generated. Basically they need to know
the grid that you are basing your tiles on (origin of the grid, size of
each tile in terms geographical extent and in terms of pixel size). And
you need to define this for every zoom level (because for a given
resolution the tiles may not cover the full extent exactly).

The result is a set of grids, aka gridset, which all your clients need
to know about and use. The most commonly used, but rarely the best, is
the one Google- and Bing Maps use for Spherical Mercator.

Not the right person to ask what raster approach is the most efficient,
but converting the JPEGs to GeoTIFFs with overviews and inner tiling
will probably be quiet fast, even though they take up more space.
Performance on that end is less important if you can use tile caching.
You can have them act as one layer using imagemosaic.

-Arne




On 12/14/12 14:21 , Jonathan Moules wrote:
> Hi List,
> A few generic questions.
>
> First: What are gridsets actually for? I know how to create them (the
> documentation is very good for that), but I can't figure out what
> they're for.
>
> Second: I have a number of large rasters in JPEG2000 format (up to
> several GB in size). GeoServer can't access these (we're not going to
> use the extension). I have FME so can do lots of processing on them if
> needed: What's the best way to serve these up? I was thinking of
> tiling them (WMS can serve tiled data right?). But then the question
> becomes, which tiling system do I use? There seem to be several:
>
> imagePyramid -
> http://docs.geoserver.org/stable/en/user/tutorials/imagepyramid/imagepyramid.html?highlight=tiling
> JDBC -
> http://docs.geoserver.org/stable/en/user/tutorials/imagemosaic-jdbc/imagemosaic-jdbc_tutorial.html?highlight=tiling
> imageMosaic -
> http://docs.geoserver.org/stable/en/user/tutorials/image_mosaic_plugin/imagemosaic.html
>
> What are the advantages/disadvantages of each? I'm guessing some are
> easier to create/use and possibly faster.
>
> Thanks,
> Jonathan
>
>
> This transmission is intended for the named addressee(s) only and may
> contain sensitive or protectively marked material up to RESTRICTED and
> should be handled accordingly. Unless you are the named addressee (or
> authorised to receive it for the addressee) you may not copy or use
> it, or disclose it to anyone else. If you have received this
> transmission in error please notify the sender immediately. All email
> traffic sent to or from us, including without limitation all GCSX
> traffic, may be subject to recording and/or monitoring in accordance
> with relevant legislation.
>
>
> ------------------------------------------------------------------------------
> LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
> Remotely access PCs and mobile devices and provide instant support
> Improve your efficiency, and focus on delivering more value-add services
> Discover what IT Professionals Know. Rescue delivers
> http://p.sf.net/sfu/logmein_12329d2d
>
>
> _______________________________________________
> Geoserver-users mailing list
> Geoserver-***@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-users

[Jonathan Moules]

Hi Arne,
Thanks for the reply. I get what gridsets do (further clarified by your
reply), I don't understand how to use them. What other parts of GeoServer
do they interact with? Or what parts of the client software? And how do you
get the client software to use the same gridsets? The documentation only
tells me how to create them, it doesn't say what I should do after that.


-----

Unfortunately FME can't create GeoTIFFs with overviews and possibly not
inner tiling too. I'm aware GDAL does but we'd prefer to use FME.
Does anyone else have any suggestions how to process the rasters?
Cheers,
Jonathan



On 16 December 2012 23:23, Arne Kepp <***@tiledmarble.org> wrote:

> If you want to have efficient caching it is crucial that clients ask for
> the tiles that you have already generated. Basically they need to know the
> grid that you are basing your tiles on (origin of the grid, size of each
> tile in terms geographical extent and in terms of pixel size). And you need
> to define this for every zoom level (because for a given resolution the
> tiles may not cover the full extent exactly).
>
> The result is a set of grids, aka gridset, which all your clients need to
> know about and use. The most commonly used, but rarely the best, is the one
> Google- and Bing Maps use for Spherical Mercator.
>
> Not the right person to ask what raster approach is the most efficient,
> but converting the JPEGs to GeoTIFFs with overviews and inner tiling will
> probably be quiet fast, even though they take up more space. Performance on
> that end is less important if you can use tile caching. You can have them
> act as one layer using imagemosaic.
>
> -Arne
>
>
>
>
>
> On 12/14/12 14:21 , Jonathan Moules wrote:
>
> Hi List,
> A few generic questions.
>
> First: What are gridsets actually for? I know how to create them (the
> documentation is very good for that), but I can't figure out what they're
> for.
>
> Second: I have a number of large rasters in JPEG2000 format (up to
> several GB in size). GeoServer can't access these (we're not going to use
> the extension). I have FME so can do lots of processing on them if needed:
> What's the best way to serve these up? I was thinking of tiling them (WMS
> can serve tiled data right?). But then the question becomes, which tiling
> system do I use? There seem to be several:
>
> imagePyramid -
> http://docs.geoserver.org/stable/en/user/tutorials/imagepyramid/imagepyramid.html?highlight=tiling
> JDBC -
> http://docs.geoserver.org/stable/en/user/tutorials/imagemosaic-jdbc/imagemosaic-jdbc_tutorial.html?highlight=tiling
> imageMosaic -
> http://docs.geoserver.org/stable/en/user/tutorials/image_mosaic_plugin/imagemosaic.html
>
> What are the advantages/disadvantages of each? I'm guessing some are
> easier to create/use and possibly faster.
>
> Thanks,
> Jonathan
>
>
> This transmission is intended for the named addressee(s) only and may
> contain sensitive or protectively marked material up to RESTRICTED and
> should be handled accordingly. Unless you are the named addressee (or
> authorised to receive it for the addressee) you may not copy or use it, or
> disclose it to anyone else. If you have received this transmission in error
> please notify the sender immediately. All email traffic sent to or from us,
> including without limitation all GCSX traffic, may be subject to recording
> and/or monitoring in accordance with relevant legislation.
>
>
> ------------------------------------------------------------------------------
> LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
> Remotely access PCs and mobile devices and provide instant support
> Improve your efficiency, and focus on delivering more value-add services
> Discover what IT Professionals Know. Rescue delivershttp://p.sf.net/sfu/logmein_12329d2d
>
>
>
> ________________________
_______________________
> Geoserver-users mailing listGeoserver-***@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/geoserver-users
>
>
>


This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

[Rahkonen Jukka]

Hi,

The idea is that the GetCapabilities message includes all the necessary information about the configured gridsets which are supported by the server. After that It is up to the client software or programmer to know what to do next. For example uDig knows automatically how to deal with WMS-C service once it has read the GetCapabilities. I am remembering that QGIS can do the same. With OpenLayers the developer needs to know how to make the application to use exactly the same grid system than the server. Information delivered by WMS-C, TMS and WMTS GetCapabilities is enough for that. For the Web Mercator system configuring OpenLayers is easier
http://docs.openlayers.org/library/spherical_mercator.html

Once you have created the gridset and layers are available from the service you have finished your work on the Geoserver/GWS side. Next you will need to start reading the client software documents and learning how to utilize those services.

FME seems to be able to use inner tiling http://docs.safe.com/fme/reader_writerPDF/geotiff.pdf

If you want definitely to use FME you can create the tiled tiffs with that and handle just the overviews/pyramid layers with GDAL by running gdaladdo for the images. Better alternative for Geoserver could be to use gdal_retile http://docs.geoserver.org/stable/en/user/tutorials/imagepyramid/imagepyramid.html

-Jukka Rahkonen-

Jonathan Moules wrote:

Hi Arne,
Thanks for the reply. I get what gridsets do (further clarified by your reply), I don't understand how to use them. What other parts of GeoServer do they interact with? Or what parts of the client software? And how do you get the client software to use the same gridsets? The documentation only tells me how to create them, it doesn't say what I should do after that.


-----

Unfortunately FME can't create GeoTIFFs with overviews and possibly not inner tiling too. I'm aware GDAL does but we'd prefer to use FME.
Does anyone else have any suggestions how to process the rasters?
Cheers,
Jonathan


On 16 December 2012 23:23, Arne Kepp <***@tiledmarble.org<mailto:***@tiledmarble.org>> wrote:
If you want to have efficient caching it is crucial that clients ask for the tiles that you have already generated. Basically they need to know the grid that you are basing your tiles on (origin of the grid, size of each tile in terms geographical extent and in terms of pixel size). And you need to define this for every zoom level (because for a given resolution the tiles may not cover the full extent exactly).

The result is a set of grids, aka gridset, which all your clients need to know about and use. The most commonly used, but rarely the best, is the one Google- and Bing Maps use for Spherical Mercator.

Not the right person to ask what raster approach is the most efficient, but converting the JPEGs to GeoTIFFs with overviews and inner tiling will probably be quiet fast, even though they take up more space. Performance on that end is less important if you can use tile caching. You can have them act as one layer using imagemosaic.

-Arne





On 12/14/12 14:21 , Jonathan Moules wrote:
Hi List,
A few generic questions.

First: What are gridsets actually for? I know how to create them (the documentation is very good for that), but I can't figure out what they're for.

Second: I have a number of large rasters in JPEG2000 format (up to several GB in size). GeoServer can't access these (we're not going to use the extension). I have FME so can do lots of processing on them if needed: What's the best way to serve these up? I was thinking of tiling them (WMS can serve tiled data right?). But then the question becomes, which tiling system do I use? There seem to be several:

imagePyramid - http://docs.geoserver.org/stable/en/user/tutorials/imagepyramid/imagepyramid.html?highlight=tiling
JDBC - http://docs.geoserver.org/stable/en/user/tutorials/imagemosaic-jdbc/imagemosaic-jdbc_tutorial.html?highlight=tiling
imageMosaic - http://docs.geoserver.org/stable/en/user/tutorials/image_mosaic_plugin/imagemosaic.html

What are the advantages/disadvantages of each? I'm guessing some are easier to create/use and possibly faster.

Thanks,
Jonathan

This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.


------------------------------------------------------------------------------

LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial

Remotely access PCs and mobile devices and provide instant support

Improve your efficiency, and focus on delivering more value-add services

Discover what IT Professionals Know. Rescue delivers

http://p.sf.net/sfu/logmein_12329d2d


________________

_________________________

______

Geoserver-users mailing list

Geoserver-***@lists.sourceforge.net<mailto:Geoserver-***@lists.sourceforge.net>https://lists.sourceforge.net/lists/listinfo/geoserver-users




This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

[Jonathan Moules]

Hi Jukka,
Thanks for the information.

Relating to the TIFF's, I'd still prefer to use one single tool
(specifically FME) for tile/image creation. FME can create pyramids, but
not as part of a GeoTIFF. So I can create separate TIFF files and tile them
if necessary too (either internal tiling or regular tiling), but how do I
get these into GeoServer?

Is there a resource out there which describes the different Raster serving
extensions and the advantages/disadvantages of each?

Cheers,
Jonathan


On 19 December 2012 11:15, Rahkonen Jukka <***@mmmtike.fi> wrote:

> Hi,****
>
> ** **
>
> The idea is that the GetCapabilities message includes all the necessary
> information about the configured gridsets which are supported by the
> server. After that It is up to the client software or programmer to know
> what to do next. For example uDig knows automatically how to deal with
> WMS-C service once it has read the GetCapabilities. I am remembering that
> QGIS can do the same. With OpenLayers the developer needs to know how to
> make the application to use exactly the same grid system than the server.
> Information delivered by WMS-C, TMS and WMTS GetCapabilities is enough for
> that. For the Web Mercator system configuring OpenLayers is easier****
>
> http://docs.openlayers.org/library/spherical_mercator.html****
>
> ** **
>
> Once you have created the gridset and layers are available from the
> service you have finished your work on the Geoserver/GWS side. Next you
> will need to start reading the client software documents and learning how
> to utilize those services. ****
>
> ** **
>
> FME seems to be able to use inner tiling
> http://docs.safe.com/fme/reader_writerPDF/geotiff.pdf****
>
> ** **
>
> If you want definitely to use FME you can create the tiled tiffs with that
> and handle just the overviews/pyramid layers with GDAL by running gdaladdo
> for the images. Better alternative for Geoserver could be to use
> gdal_retile
> http://docs.geoserver.org/stable/en/user/tutorials/imagepyramid/imagepyramid.html
> ****
>
> ** **
>
> -Jukka Rahkonen-****
>
> ** **
>
> Jonathan Moules wrote:****
>
> ** **
>
> Hi Arne,****
>
> Thanks for the reply. I get what gridsets do (further clarified by your
> reply), I don't understand how to use them. What other parts of GeoServer
> do they interact with? Or what parts of the client software? And how do you
> get the client software to use the same gridsets? The documentation only
> tells me how to create them, it doesn't say what I should do after that.**
> **
>
> ** **
>
> ** **
>
> -----****
>
> ** **
>
> Unfortunately FME can't create GeoTIFFs with overviews and possibly not
> inner tiling too. I'm aware GDAL does but we'd prefer to use FME.****
>
> Does anyone else have any suggestions how to process the rasters?****
>
> Cheers,****
>
> Jonathan****
>
>
>
> ****
>
> On 16 December 2012 23:23, Arne Kepp <***@tiledmarble.org> wrote:****
>
> If you want to have efficient caching it is crucial that clients ask for
> the tiles that you have already generated. Basically they need to know the
> grid that you are basing your tiles on (origin of the grid, size of each
> tile in terms geographical extent and in terms of pixel size). And you need
> to define this for every zoom level (because for a given resolution the
> tiles may not cover the full extent exactly).
>
> The result is a set of grids, aka gridset, which all your clients need to
> know about and use. The most commonly used, but rarely the best, is the one
> Google- and Bing Maps use for Spherical Mercator.
>
> Not the right person to ask what raster approach is the most efficient,
> but converting the JPEGs to GeoTIFFs with overviews and inner tiling will
> probably be quiet fast, even though they take up more space. Performance on
> that end is less important if you can use tile caching. You can have them
> act as one layer using imagemosaic.
>
> -Arne****
>
>
>
>
>
>
> On 12/14/12 14:21 , Jonathan Moules wrote:****
>
> Hi List,****
>
> A few generic questions.****
>
> ** **
>
> First: What are gridsets actually for? I know how to create them (the
> documentation is very good for that), but I can't figure out what they're
> for.****
>
> ** **
>
> Second: I have a number of large rasters in JPEG2000 format (up to several
> GB in size). GeoServer can't access these (we're not going to use the
> extension). I have FME so can do lots of processing on them if needed:
> What's the best way to serve these up? I was thinking of tiling them (WMS
> can serve tiled data right?). But then the question becomes, which tiling
> system do I use? There seem to be several:****
>
> ** **
>
> imagePyramid -
> http://docs.geoserver.org/stable/en/user/tutorials/imagepyramid/imagepyramid.html?highlight=tiling
> ****
>
> JDBC -
> http://docs.geoserver.org/stable/en/user/tutorials/imagemosaic-jdbc/imagemosaic-jdbc_tutorial.html?highlight=tiling
> ****
>
> imageMosaic -
> http://docs.geoserver.org/stable/en/user/tutorials/image_mosaic_plugin/imagemosaic.html
> ****
>
> ** **
>
> What are the advantages/disadvantages of each? I'm guessing some are
> easier to create/use and possibly faster.****
>
> ** **
>
> Thanks,****
>
> Jonathan****
>
> ** **
>
> This transmission is intended for the named addressee(s) only and may
> contain sensitive or protectively marked material up to RESTRICTED and
> should be handled accordingly. Unless you are the named addressee (or
> authorised to receive it for the addressee) you may not copy or use it, or
> disclose it to anyone else. If you have received this transmission in error
> please notify the sender immediately. All email traffic sent to or from us,
> including without limitation all GCSX traffic, may be subject to recording
> and/or monitoring in accordance with relevant legislation.
>
> ****
>
> ------------------------------------------------------------------------------****
>
> LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial****
>
> Remotely access PCs and mobile devices and provide instant support****
>
> Improve your efficiency, and focus on delivering more value-add services****
>
> Discover what IT Professionals Know. Rescue delivers****
>
> http://p.sf.net/sfu/logmein_12329d2d****
>
> ** **
>
> ________________****
>
> ________________________
_****
>
> ______****
>
> Geoserver-users mailing list****
>
> Geoserver-***@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/geoserver-users****
>
> ** **
>
> ** **
>
>
>
> This transmission is intended for the named addressee(s) only and may
> contain sensitive or protectively marked material up to RESTRICTED and
> should be handled accordingly. Unless you are the named addressee (or
> authorised to receive it for the addressee) you may not copy or use it, or
> disclose it to anyone else. If you have received this transmission in error
> please notify the sender immediately. All email traffic sent to or from us,
> including without limitation all GCSX traffic, may be subject to recording
> and/or monitoring in accordance with relevant legislation.****
>
>
> ------------------------------------------------------------------------------
> LogMeIn Rescue: Anywhere, Anytime Remote support for IT. Free Trial
> Remotely access PCs and mobile devices and provide instant support
> Improve your efficiency, and focus on delivering more value-add services
> Discover what IT Professionals Know. Rescue delivers
> http://p.sf.net/sfu/logmein_12329d2d
> ________________________
_______________________
> Geoserver-users mailing list
> Geoserver-***@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-users
>
>


This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

[Andrea Aime]

On Thu, Dec 20, 2012 at 12:16 PM, Jonathan Moules <
***@warwickshire.gov.uk> wrote:

> Hi Jukka,
> Thanks for the information.
>
> Relating to the TIFF's, I'd still prefer to use one single tool
> (specifically FME) for tile/image creation. FME can create pyramids, but
> not as part of a GeoTIFF. So I can create separate TIFF files and tile them
> if necessary too (either internal tiling or regular tiling), but how do I
> get these into GeoServer?
>
> Is there a resource out there which describes the different Raster serving
> extensions and the advantages/disadvantages of each?
>

Here:
http://demo.geo-solutions.it/share/foss4g2011/gs_steroids_sgiannec_foss4g2011.pdf

Cheers
Andrea

--
==
Our support, Your Success! Visit http://opensdi.geo-solutions.it for more
information.
==

Ing. Andrea Aime
@geowolf
Technical Lead

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: [phone]
fax: [phone]
mob: [phone]

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------

[Jonathan Moules]

Thanks Andrea, this looks like a very useful document although it is
looking like I'm not going to be able to do this with FME from what I can
see.
Cheers,
Jonathan



On 20 December 2012 21:06, Andrea Aime <***@geo-solutions.it> wrote:

> On Thu, Dec 20, 2012 at 12:16 PM, Jonathan Moules <
> ***@warwickshire.gov.uk> wrote:
>
>> Hi Jukka,
>> Thanks for the information.
>>
>> Relating to the TIFF's, I'd still prefer to use one single tool
>> (specifically FME) for tile/image creation. FME can create pyramids, but
>> not as part of a GeoTIFF. So I can create separate TIFF files and tile them
>> if necessary too (either internal tiling or regular tiling), but how do I
>> get these into GeoServer?
>>
>> Is there a resource out there which describes the different Raster
>> serving extensions and the advantages/disadvantages of each?
>>
>
> Here:
> http://demo.geo-solutions.it/share/foss4g2011/gs_steroids_sgiannec_foss4g2011.pdf
>
> Cheers
> Andrea
>
> --
> ==
> Our support, Your Success! Visit http://opensdi.geo-solutions.it for more
> information.
> ==
>
> Ing. Andrea Aime
> @geowolf
> Technical Lead
>
> GeoSolutions S.A.S.
> Via Poggio alle Viti 1187
> 55054 Massarosa (LU)
> Italy
> phone: [phone]
> fax: [phone]
> mob: [phone]
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> -------------------------------------------------------
>


This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

[Jonathan Moules]

Hi list,
Following this up, I've decided to go the GeoTIFF/BigTiff route with inner
tiling and overviews based on Andrea's document.
But unfortunately there doesn't seem to be any documentation on the
GeoServer pages about this stuff. I've never used GDAL before and generally
avoid the command line (there's a reason we're trying for GeoServer not
MapServer ;-) ) so am not sure where to start.

My source input is hundreds/thousands of LZW compressed GeoTIFF tiles.
These need to be mosaiced together (I can do that in FME easily enough).
But then what? Do I compress them at this point? Or inner tiling? Or
Overview? Does it matter what order these things are done?
And what settings should I use? For instance Andrea's document gives a
"blocksize" of 512, but Jukka's comments were on the order of 10,000 and
the default is 256.

Is there a tutorial out there on this I've failed to find?
Thanks,
Jonathan


On 21 December 2012 14:45, Jonathan Moules <
***@warwickshire.gov.uk> wrote:

> Thanks Andrea, this looks like a very useful document although it is
> looking like I'm not going to be able to do this with FME from what I can
> see.
> Cheers,
> Jonathan
>
>
>
>
> On 20 December 2012 21:06, Andrea Aime <***@geo-solutions.it>wrote:
>
>> On Thu, Dec 20, 2012 at 12:16 PM, Jonathan Moules <
>> ***@warwickshire.gov.uk> wrote:
>>
>>> Hi Jukka,
>>> Thanks for the information.
>>>
>>> Relating to the TIFF's, I'd still prefer to use one single tool
>>> (specifically FME) for tile/image creation. FME can create pyramids, but
>>> not as part of a GeoTIFF. So I can create separate TIFF files and tile them
>>> if necessary too (either internal tiling or regular tiling), but how do I
>>> get these into GeoServer?
>>>
>>> Is there a resource out there which describes the different Raster
>>> serving extensions and the advantages/disadvantages of each?
>>>
>>
>> Here:
>> http://demo.geo-solutions.it/share/foss4g2011/gs_steroids_sgiannec_foss4g2011.pdf
>>
>> Cheers
>> Andrea
>>
>> --
>> ==
>> Our support, Your Success! Visit http://opensdi.geo-solutions.it for
>> more information.
>> ==
>>
>> Ing. Andrea Aime
>> @geowolf
>> Technical Lead
>>
>> GeoSolutions S.A.S.
>> Via Poggio alle Viti 1187
>> 55054 Massarosa (LU)
>> Italy
>> phone: [phone]
>> fax: [phone]
>> mob: [phone]
>>
>> http://www.geo-solutions.it
>> http://twitter.com/geosolutions_it
>>
>> -------------------------------------------------------
>>
>
>


This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

[Jonathan Moules]

*Bump* Anyone?
This seems like the sort of thing that'd be well documented, but my
google-fu seems to have failed me.
The closest I can get is a presentation about using the ImagePyramid plugin
(
http://www.slideshare.net/geosolutions/creating-a-pyramid-with-gdal-reltile-for-serving-with-geoserverand
http://geo-solutions.blogspot.ru/2009/11/preparing-pyramid-for-geoserver-with.html
)
Cheers,
Jonathan


On 27 December 2012 13:20, Jonathan Moules <
***@warwickshire.gov.uk> wrote:

> Hi list,
> Following this up, I've decided to go the GeoTIFF/BigTiff route with inner
> tiling and overviews based on Andrea's document.
> But unfortunately there doesn't seem to be any documentation on the
> GeoServer pages about this stuff. I've never used GDAL before and generally
> avoid the command line (there's a reason we're trying for GeoServer not
> MapServer ;-) ) so am not sure where to start.
>
> My source input is hundreds/thousands of LZW compressed GeoTIFF tiles.
> These need to be mosaiced together (I can do that in FME easily enough).
> But then what? Do I compress them at this point? Or inner tiling? Or
> Overview? Does it matter what order these things are done?
> And what settings should I use? For instance Andrea's document gives a
> "blocksize" of 512, but Jukka's comments were on the order of 10,000 and
> the default is 256.
>
> Is there a tutorial out there on this I've failed to find?
> Thanks,
> Jonathan
>
>
> On 21 December 2012 14:45, Jonathan Moules <
> ***@warwickshire.gov.uk> wrote:
>
>> Thanks Andrea, this looks like a very useful document although it is
>> looking like I'm not going to be able to do this with FME from what I can
>> see.
>> Cheers,
>> Jonathan
>>
>>
>>
>>
>> On 20 December 2012 21:06, Andrea Aime <***@geo-solutions.it>wrote:
>>
>>> On Thu, Dec 20, 2012 at 12:16 PM, Jonathan Moules <
>>> ***@warwickshire.gov.uk> wrote:
>>>
>>>> Hi Jukka,
>>>> Thanks for the information.
>>>>
>>>> Relating to the TIFF's, I'd still prefer to use one single tool
>>>> (specifically FME) for tile/image creation. FME can create pyramids, but
>>>> not as part of a GeoTIFF. So I can create separate TIFF files and tile them
>>>> if necessary too (either internal tiling or regular tiling), but how do I
>>>> get these into GeoServer?
>>>>
>>>> Is there a resource out there which describes the different Raster
>>>> serving extensions and the advantages/disadvantages of each?
>>>>
>>>
>>> Here:
>>> http://demo.geo-solutions.it/share/foss4g2011/gs_steroids_sgiannec
_foss4g2011.pdf
>>>
>>> Cheers
>>> Andrea
>>>
>>> --
>>> ==
>>> Our support, Your Success! Visit http://opensdi.geo-solutions.it for
>>> more information.
>>> ==
>>>
>>> Ing. Andrea Aime
>>> @geowolf
>>> Technical Lead
>>>
>>> GeoSolutions S.A.S.
>>> Via Poggio alle Viti 1187
>>> 55054 Massarosa (LU)
>>> Italy
>>> phone: [phone]
>>> fax: [phone]
>>> mob: [phone]
>>>
>>> http://www.geo-solutions.it
>>> http://twitter.com/geosolutions_it
>>>
>>> -------------------------------------------------------
>>>
>>
>>
>


This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

[Russell Hore]

This might not be exactly what you want but my workflow (Under Linux) is

1) Take the raw TIF images
2) Add a .PRJ file for each image
3) Add the World file for each
4) Run gdal_translate for each file
for file in `ls *tif`
do
gdal_translate -of GTiff -co "TILED=YES" -co "COMPRESS=DEFLATE" -co "BLOCKXSIZE=512" -co "BLOCKYSIZE=512" -a_srs "EPSG:27700" $file out/$file
done
5) Ran a gdaladdo for each file in 'out'
for file in `ls *tif`
do
gdaladdo -r average $file 1 2 4 8 16 32 64 128
done

6) Added them to Geoserver (2.2 RC1) using ImageMosaic

Russ

On 31 Dec 2012, at 10:44, Jonathan Moules <***@warwickshire.gov.uk> wrote:

> *Bump* Anyone?
> This seems like the sort of thing that'd be well documented, but my google-fu seems to have failed me.
> The closest I can get is a presentation about using the ImagePyramid plugin ( http://www.slideshare.net/geosolutions/creating-a-pyramid-with-gdal-reltile-for-serving-with-geoserver and http://geo-solutions.blogspot.ru/2009/11/preparing-pyramid-for-geoserver-with.html )
> Cheers,
> Jonathan
>
>
> On 27 December 2012 13:20, Jonathan Moules <***@warwickshire.gov.uk> wrote:
> Hi list,
> Following this up, I've decided to go the GeoTIFF/BigTiff route with inner tiling and overviews based on Andrea's document.
> But unfortunately there doesn't seem to be any documentation on the GeoServer pages about this stuff. I've never used GDAL before and generally avoid the command line (there's a reason we're trying for GeoServer not MapServer ;-) ) so am not sure where to start.
>
> My source input is hundreds/thousands of LZW compressed GeoTIFF tiles. These need to be mosaiced together (I can do that in FME easily enough).
> But then what? Do I compress them at this point? Or inner tiling? Or Overview? Does it matter what order these things are done?
> And what settings should I use? For instance Andrea's document gives a "blocksize" of 512, but Jukka's comments were on the order of 10,000 and the default is 256.
>
> Is there a tutorial out there on this I've failed to find?
> Thanks,
> Jonathan
>
>
> On 21 December 2012 14:45, Jonathan Moules <***@warwickshire.gov.uk> wrote:
> Thanks Andrea, this looks like a very useful document although it is looking like I'm not going to be able to do this with FME from what I can see.
> Cheers,
> Jonathan
>
>
>
>
> On 20 December 2012 21:06, Andrea Aime <***@geo-solutions.it> wrote:
> On Thu, Dec 20, 2012 at 12:16 PM, Jonathan Moules <***@warwickshire.gov.uk> wrote:
> Hi Jukka,
> Thanks for the information.
>
> Relating to the TIFF's, I'd still prefer to use one single tool (specifically FME) for tile/image creation. FME can create pyramids, but not as part of a GeoTIFF. So I can create separate TIFF files and tile them if necessary too (either internal tiling or regular tiling), but how do I get these into GeoServer?
>
> Is there a resource out there which describes the different Raster serving extensions and the advantages/disadvantages of each?
>
> Here: http://demo.geo-solutions.it/share/foss4g2011/gs_steroids_sgiannec_foss4g2011.pdf
>
> Cheers
> Andrea
>
> --
> ==
> Our support, Your Success! Visit http://opensdi.geo-solutions.it for more information.
> ==
>
> Ing. Andrea Aime
> @geowolf
> Technical Lead
>
> GeoSolutions S.A.S.
> Via Poggio alle Viti 1187
> 55054 Massarosa (LU)
> Italy
> phone: [phone]
> fax: [phone]
> mob: [phone]
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> -------------------------------------------------------
>
>
>
>
>
> This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.
> ------------------------------------------------------------------------------
> Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
> MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current
> with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
> MVPs and experts. SALE $99.99 this month only -- learn more at:
> http://p.sf.net/sfu/learnmore_122412_______________________________________________
> Geoserver-users mailing list
> Geoserver-***@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-users

[Simone Giannecchini]

Ciao Jonathan,
I don't know about FME therefore I cannot comment on how to do such
things with it. I hardly believe many people on this list (as well as
on other OS oriented lists) will know it much.
This to say that most intructions you'll find will be oriented towards
OS tools like GDAL utilities rather than towards proprietary solutions
like FME.

That said, the suggestions you are looking for are inside the document
andrea suggested (here the link again http://goo.gl/TXJRS):

- slides 7-9 when do use what
- slides 10-13 how to optimize each single geotiff
- slides 17-19 how to prepare a mosaic

and so on.

If you check on the web you'll also find other docs that compare tile
sizes rather than types of compression, e.g.
http://www.fosslc.org/drupal/content/tuning-gdal-raster-performance

Regards,
Simone Giannecchini
==
Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information.
==

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: [phone]
fax: [phone]
mob: [phone]

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------


On Thu, Dec 27, 2012 at 2:20 PM, Jonathan Moules
<***@warwickshire.gov.uk> wrote:
> Hi list,
> Following this up, I've decided to go the GeoTIFF/BigTiff route with inner
> tiling and overviews based on Andrea's document.
> But unfortunately there doesn't seem to be any documentation on the
> GeoServer pages about this stuff. I've never used GDAL before and generally
> avoid the command line (there's a reason we're trying for GeoServer not
> MapServer ;-) ) so am not sure where to start.
>
> My source input is hundreds/thousands of LZW compressed GeoTIFF tiles. These
> need to be mosaiced together (I can do that in FME easily enough).
> But then what? Do I compress them at this point? Or inner tiling? Or
> Overview? Does it matter what order these things are done?
> And what settings should I use? For instance Andrea's document gives a
> "blocksize" of 512, but Jukka's comments were on the order of 10,000 and the
> default is 256.
>
> Is there a tutorial out there on this I've failed to find?
> Thanks,
> Jonathan
>
>
> On 21 December 2012 14:45, Jonathan Moules
> <***@warwickshire.gov.uk> wrote:
>>
>> Thanks Andrea, this looks like a very useful document although it is
>> looking like I'm not going to be able to do this with FME from what I can
>> see.
>> Cheers,
>> Jonathan
>>
>>
>>
>>
>> On 20 December 2012 21:06, Andrea Aime <***@geo-solutions.it>
>> wrote:
>>>
>>> On Thu, Dec 20, 2012 at 12:16 PM, Jonathan Moules
>>> <***@warwickshire.gov.uk> wrote:
>>>>
>>>> Hi Jukka,
>>>> Thanks for the information.
>>>>
>>>> Relating to the TIFF's, I'd still prefer to use one single tool
>>>> (specifically FME) for tile/image creation. FME can create pyramids, but not
>>>> as part of a GeoTIFF. So I can create separate TIFF files and tile them if
>>>> necessary too (either internal tiling or regular tiling), but how do I get
>>>> these into GeoServer?
>>>>
>>>> Is there a resource out there which describes the different Raster
>>>> serving extensions and the advantages/disadvantages of each?
>>>
>>>
>>> Here:
>>> http://demo.geo-solutions.it/share/foss4g2011/gs_steroids_sgiannec_foss4g2011.pdf
>>>
>>> Cheers
>>> Andrea
>>>
>>> --
>>> ==
>>> Our support, Your Success! Visit http://opensdi.geo-solutions.it for more
>>> information.
>>> ==
>>>
>>> Ing. Andrea Aime
>>> @geowolf
>>> Technical Lead
>>>
>>> GeoSolutions S.A.S.
>>> Via Poggio alle Viti 1187
>>> 55054 Massarosa (LU)
>>> Italy
>>> phone: [phone]
>>> fax: [phone]
>>> mob: [phone]
>>>
>>> http://www.geo-solutions.it
>>> http://twitter.com/geosolutions_it
>>>
>>> -------------------------------------------------------
>>
>>
>
>
>
> This transmission is intended for the named addressee(s) only and may
> contain sensitive or protectively marked material up to RESTRICTED and
> should be handled accordingly. Unless you are the named addressee (or
> authorised to receive it for the addressee) you may not copy or use it, or
> disclose it to anyone else. If you have received this transmission in error
> please notify the sender immediately. All email traffic sent to or from us,
> including without limitation all GCSX traffic, may be subject to recording
> and/or monitoring in accordance with relevant legislation.
>
> ------------------------------------------------------------------------------
> Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
> MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current
> with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
> MVPs and experts. ON SALE this month only -- learn more at:
> http://p.sf.net/sfu/learnmore_122712
> _______________________________________________
> Geoserver-users mailing list
> Geoserver-***@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-users
>

[Simone Giannecchini]

Ciao Jonathan,
please find my answers inline below...

Regards,
Simone Giannecchini
==
Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information.
==

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: [phone]
fax: [phone]
mob: [phone]

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------


On Wed, Jan 2, 2013 at 2:01 PM, Jonathan Moules
<***@warwickshire.gov.uk> wrote:
> Hi Simone,
> I did use pages 7-9 to decide to go for BigTIFF and so 17-19 isn't
> applicable as I'm not going to ImageMosaic.
>
> While pages 10-13 are useful, its a slideshow and so lacks details which
> were probably included in the spoken part of the presentation. Its the
> details that I'm looking for. What tile size is best? What resampling
> algorithm? What overview size/scales? Etc.
> Don't get me wrong, its an excellent document, but it only provides an
> overview so can't substitute for actual detailed documentation.
>
> Your second link answers the tile-size question, but specific for MapServer
> (i.e. 256).
>
> Also, page 11 hints at "external overviews" but there's no indication as to
> how to create them or use them with GeoServer. FME can create them as I'm
> sure GDAL can too, but again, I don't know how to use External Overviews
> with Geoserver. A search of the docs for "external overviews" finds nothing
> and a google only finds that PDF.
> Does anyone have more information on how to use these with GeoServer?
>

No worries, actually this feedback gives me some hints on how to
update the presentation :)

That said, the considerations done for MapServer applies more or less
to GeoServer as they are more related
to how to geotiff works rather than to how MapServer or GeoServer works.

Anyway, my suggestions are as follows:
- tile size-
256 or for very large bigtiff 512 is my preferred size.

Using a tile size that is too small may result in too many seeks
operations on a side and on the other side might result into a TIFF
directory explosion.
The more tiles we have the more info we need to encode in the TIFF
Directory, which is essentially a list of offset locations that tells
us where the data is on disk.
For bigtiff where we have thousands of tiles it can be MB big.

- Resampling -
This depends on the data.
For RGB like data (orthos and the like) I would gor for higher order
interpolation to reduce aliasing (cripsy images).
For data like DEM or other data that contains real values to which you
will want to apply a color map I would for for Nearest neighbor to NOT
introduce artificial values.
In some case you might want to use bilinear of average but this
depends much on what you want to do.

- Overviews -
I usually use steps of 2, as many of them as possible although I
usually don't create overviews smaller than the tile size

- External Overviews -
This works only with pure geotiff (or bigtiff as well as the plugin is
the same). ImageMosaic does not (yet) support that.
External overviews are useful just in case you don't want to touch the
original files or in case they do not support inner overviews (which
is not the case for geotiff).

Creating external overviews can be done with gdaladdo, check this link
http://www.gdal.org/gdaladdo.html


> I'm fine with no FME specific documentation and wouldn't really expect
> any, but because I can't find things in the generic documentation (i.e., any
> of the above but especially external overviews), I can't necessarily
> implement an optimal solution with any utility, be it FME or GDAL.


Let me know if this helps. I will try to update those slides as well
as to create some additional docs.

You might even want to contribute some doc once you get to where you
needed to go :)

>
> Thanks,
> Jonathan
>
>
> On 2 January 2013 11:25, Simone Giannecchini
> <***@geo-solutions.it> wrote:
>>
>> Ciao Jonathan,
>> I don't know about FME therefore I cannot comment on how to do such
>> things with it. I hardly believe many people on this list (as well as
>> on other OS oriented lists) will know it much.
>> This to say that most intructions you'll find will be oriented towards
>> OS tools like GDAL utilities rather than towards proprietary solutions
>> like FME.
>>
>> That said, the suggestions you are looking for are inside the document
>> andrea suggested (here the link again http://goo.gl/TXJRS):
>>
>> - slides 7-9 when do use what
>> - slides 10-13 how to optimize each single geotiff
>> - slides 17-19 how to prepare a mosaic
>>
>> and so on.
>>
>> If you check on the web you'll also find other docs that compare tile
>> sizes rather than types of compression, e.g.
>> http://www.fosslc.org/drupal/content/tuning-gdal-raster-performance
>>
>> Regards,
>> Simone Giannecchini
>> ==
>> Our support, Your Success! Visit http://opensdi.geo-solutions.it for
>> more information.
>> ==
>>
>> Ing. Simone Giannecchini
>> @simogeo
>> Founder/Director
>>
>> GeoSolutions S.A.S.
>> Via Poggio alle Viti 1187
>> 55054 Massarosa (LU)
>> Italy
>> phone: [phone]
>> fax: [phone]
>> mob: [phone]
>>
>> http://www.geo-solutions.it
>> http://twitter.com/geosolutions _it
>>
>> -------------------------------------------------------
>>
>>
>> On Thu, Dec 27, 2012 at 2:20 PM, Jonathan Moules
>> <***@warwickshire.gov.uk> wrote:
>> > Hi list,
>> > Following this up, I've decided to go the GeoTIFF/BigTiff route with
>> > inner
>> > tiling and overviews based on Andrea's document.
>> > But unfortunately there doesn't seem to be any documentation on the
>> > GeoServer pages about this stuff. I've never used GDAL before and
>> > generally
>> > avoid the command line (there's a reason we're trying for GeoServer not
>> > MapServer ;-) ) so am not sure where to start.
>> >
>> > My source input is hundreds/thousands of LZW compressed GeoTIFF tiles.
>> > These
>> > need to be mosaiced together (I can do that in FME easily enough).
>> > But then what? Do I compress them at this point? Or inner tiling? Or
>> > Overview? Does it matter what order these things are done?
>> > And what settings should I use? For instance Andrea's document gives a
>> > "blocksize" of 512, but Jukka's comments were on the order of 10,000 and
>> > the
>> > default is 256.
>> >
>> > Is there a tutorial out there on this I've failed to find?
>> > Thanks,
>> > Jonathan
>> >
>> >
>> > On 21 December 2012 14:45, Jonathan Moules
>> > <***@warwickshire.gov.uk> wrote:
>> >>
>> >> Thanks Andrea, this looks like a very useful document although it is
>> >> looking like I'm not going to be able to do this with FME from what I
>> >> can
>> >> see.
>> >> Cheers,
>> >> Jonathan
>> >>
>> >>
>> >>
>> >>
>> >> On 20 December 2012 21:06, Andrea Aime <***@geo-solutions.it>
>> >> wrote:
>> >>>
>> >>> On Thu, Dec 20, 2012 at 12:16 PM, Jonathan Moules
>> >>> <***@warwickshire.gov.uk> wrote:
>> >>>>
>> >>>> Hi Jukka,
>> >>>> Thanks for the information.
>> >>>>
>> >>>> Relating to the TIFF's, I'd still prefer to use one single tool
>> >>>> (specifically FME) for tile/image creation. FME can create pyramids,
>> >>>> but not
>> >>>> as part of a GeoTIFF. So I can create separate TIFF files and tile
>> >>>> them if
>> >>>> necessary too (either internal tiling or regular tiling), but how do
>> >>>> I get
>> >>>> these into GeoServer?
>> >>>>
>> >>>> Is there a resource out there which describes the different Raster
>> >>>> serving extensions and the advantages/disadvantages of each?
>> >>>
>> >>>
>> >>> Here:
>> >>>
>> >>> http://demo.geo-solutions.it/share/foss4g2011/gs_steroids_sgiannec_foss4g2011.pdf
>> >>>
>> >>> Cheers
>> >>> Andrea
>> >>>
>> >>> --
>> >>> ==
>> >>> Our support, Your Success! Visit http://opensdi.geo-solutions.it for
>> >>> more
>> >>> information.
>> >>> ==
>> >>>
>> >>> Ing. Andrea Aime
>> >>> @geowolf
>> >>> Technical Lead
>> >>>
>> >>> GeoSolutions S.A.S.
>> >>> Via Poggio alle Viti 1187
>> >>> 55054 Massarosa (LU)
>> >>> Italy
>> >>> phone: [phone]
>> >>> fax: [phone]
>> >>> mob: [phone]
>> >>>
>> >>> http://www.geo-solutions.it
>> >>> http://twitter.com/geosolutions_it
>> >>>
>> >>> -------------------------------------------------------
>> >>
>> >>
>> >
>> >
>> >
>> > This transmission is intended for the named addressee(s) only and may
>> > contain sensitive or protectively marked material up to RESTRICTED and
>> > should be handled accordingly. Unless you are the named addressee (or
>> > authorised to receive it for the addressee) you may not copy or use it,
>> > or
>> > disclose it to anyone else. If you have received this transmission in
>> > error
>> > please notify the sender immediately. All email traffic sent to or from
>> > us,
>> > including without limitation all GCSX traffic, may be subject to
>> > recording
>> > and/or monitoring in accordance with relevant legislation.
>> >
>> >
>> > ------------------------------------------------------------------------------
>> > Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
>> > MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current
>> > with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
>> > MVPs and experts. ON SALE this month only -- learn more at:
>> > http://p.sf.net/sfu/learnmore_122712
>> > ______________________ _________________________
>> > Geoserver-users mailing list
>> > Geoserver-***@lists.sourceforge.net
>> > https://lists.sourceforge.net/lists/listinfo/geoserver-users
>> >
>
>
>
>
> This transmission is intended for the named addressee(s) only and may
> contain sensitive or protectively marked material up to RESTRICTED and
> should be handled accordingly. Unless you are the named addressee (or
> authorised to receive it for the addressee) you may not copy or use it, or
> disclose it to anyone else. If you have received this transmission in error
> please notify the sender immediately. All email traffic sent to or from us,
> including without limitation all GCSX traffic, may be subject to recording
> and/or monitoring in accordance with relevant legislation.

[Jonathan Moules]

Hey Simone,

Thanks for your thorough reply. Useful!

- Resampling:
I've done some experimenting with Resampling since my email and, at least
for the data I was testing, Nearest Neighbour came out best (though
obviously entirely subjective). This was for a regular RGB map (Ordnance
Survey MiniScale specifically). There was also a significant difference in
file sizes, with Nearest Neighbour being the smallest, and Cubic being the
largest (double!). I didn't expect such a significant difference, may be
worth noting somewhere.

- External Overviews:
Thanks, but I'd found out how to create them with GDAL. My issue is - how
do I get GeoServer to use them? Does it just pick them up automatically or
do I need some sort of particular directory structure?
GDAL seems to create a single, absurdly huge file, so I guess it doesn't
compress the external overviews. Based on what I'm seeing, I'm not sure if
FME can create them; I expected multiple separate files (one for each
layer), but the "external overviews" are basically identical to internal
overviews but minus the source data.

-Overviews:
How do you determine how many levels to create? I remember this page from
years ago when last I used GeoServer:
http://docs.geoserver.org/stable/en/user/tutorials/imagemosaic-jdbc/imagemosaic-jdbc_tutorial.html#how-many-pyramids-are-needed-
does this carry over to GeoTIFF's overviews?
I may be reading this wrong, but the example in the PDF has 7 layers but
only really needs about 3 as after that they're smaller than the 1:1 tiles.

Is there any point in creating pyramids with a value of 1 (as in, 1 2 4
8...) ? GDAL lets you do this but if I'm understanding this correctly,
they're entirely superfluous (I mention it because Russ's example he
generously posted starts at 1).


Tile size:
Might there be a formula for calculating optimal size?
For example, if I have a 94488*157480 pixel GeoTIFF (my largest - about
2.5GB compressed), if I divide them by 512 I get
184*307 tiles = 56,488 tiles

For 256 I get:
368*614 tiles = 225,952 tiles

So if there is/was a formula, it might be easier to figure out what size to
go with, or at what point you'd want to be scaling up to ImageMosaic.
Based on disk seeks etc, and some experimentation, I'd guess that a number
could be contrived allowing the documentation to say:
"If you end up with > X,000 tiles, go with ImageMosiac" and "Aim for
between X,000 and X0,000 tiles in a GeoTIFF"
Just a thought, would be helpful for optimisation.

Documentation:
I was considering contributing to it for this but at this point certainly
don't know enough (you may have noticed from my above questions ;-) ).
I would however suggest that this section of the help:
http://docs.geoserver.org/stable/en/user/production/data.html?highlight=gdaladdo#pick-the-best-performing-coverage-formats-
It would gain from the info on pages 7-9 of your PDF.

Cheers!
Jonathan


On 2 January 2013 15:03, Simone Giannecchini <
***@geo-solutions.it> wrote:

> Ciao Jonathan,
> please find my answers inline below...
>
> Regards,
> Simone Giannecchini
> ==
> Our support, Your Success! Visit http://opensdi.geo-solutions.it for
> more information.
> ==
>
> Ing. Simone Giannecchini
> @simogeo
> Founder/Director
>
> GeoSolutions S.A.S.
> Via Poggio alle Viti 1187
> 55054 Massarosa (LU)
> Italy
> phone: [phone]
> fax: [phone]
> mob: [phone]
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> -------------------------------------------------------
>
>
> On Wed, Jan 2, 2013 at 2:01 PM, Jonathan Moules
> <***@warwickshire.gov.uk> wrote:
> > Hi Simone,
> > I did use pages 7-9 to decide to go for BigTIFF and so 17-19 isn't
> > applicable as I'm not going to ImageMosaic.
> >
> > While pages 10-13 are useful, its a slideshow and so lacks details which
> > were probably included in the spoken part of the presentation. Its the
> > details that I'm looking for. What tile size is best? What resampling
> > algorithm? What overview size/scales? Etc.
> > Don't get me wrong, its an excellent document, but it only provides an
> > overview so can't substitute for actual detailed documentation.
> >
> > Your second link answers the tile-size question, but specific for
> MapServer
> > (i.e. 256).
> >
> > Also, page 11 hints at "external overviews" but there's no indication as
> to
> > how to create them or use them with GeoServer. FME can create them as I'm
> > sure GDAL can too, but again, I don't know how to use External Overviews
> > with Geoserver. A search of the docs for "external overviews" finds
> nothing
> > and a google only finds that PDF.
> > Does anyone have more information on how to use these with GeoServer?
> >
>
> No worries, actually this feedback gives me some hints on how to
> update the presentation :)
>
> That said, the considerations done for MapServer applies more or less
> to GeoServer as they are more related
> to how to geotiff works rather than to how MapServer or GeoServer works.
>
> Anyway, my suggestions are as follows:
> - tile size-
> 256 or for very large bigtiff 512 is my preferred size.
>
> Using a tile size that is too small may result in too many seeks
> operations on a side and on the other side might result into a TIFF
> directory explosion.
> The more tiles we have the more info we need to encode in the TIFF
> Directory, which is essentially a list of offset locations that tells
> us where the data is on disk.
> For bigtiff where we have thousands of tiles it can be MB big.
>
> - Resampling -
> This depends on the data.
> For RGB like data (orthos and the like) I would gor for higher order
> interpolation to reduce aliasing (cripsy images).
> For data like DEM or other data that contains real values to which you
> will want to apply a color map I would for for Nearest neighbor to NOT
> introduce artificial values.
> In some case you might want to use bilinear of average but this
> depends much on what you want to do.
>
> - Overviews -
> I usually use steps of 2, as many of them as possible although I
> usually don't create overviews smaller than the tile size
>
> - External Overviews -
> This works only with pure geotiff (or bigtiff as well as the plugin is
> the same). ImageMosaic does not (yet) support that.
> External overviews are useful just in case you don't want to touch the
> original files or in case they do not support inner overviews (which
> is not the case for geotiff).
>
> Creating external overviews can be done with gdaladdo, check this link
> http://www.gdal.org/gdaladdo.html
>
>
> > I'm fine with no FME specific documentation and wouldn't really expect
> > any, but because I can't find things in the generic documentation (i.e.,
> any
> > of the above but especially external overviews), I can't necessarily
> > implement an optimal solution with any utility, be it FME or GDAL.
>
>
> Let me know if this helps. I will try to update those slides as well
> as to create some additional docs.
>
> You might even want to contribute some doc once you get to where you
> needed to go :)
>
> >
> > Thanks,
> > Jonathan
> >
> >
> > On 2 January 2013 11:25, Simone Giannecchini
> > <***@geo-solutions.it> wrote:
> >>
> >> Ciao Jonathan,
> >> I don't know about FME therefore I cannot comment on how to do such
> >> things with it. I hardly believe many people on this list (as well as
> >> on other OS oriented lists) will know it much.
> >> This to say that most intructions you'll find will be oriented towards
> >> OS tools like GDAL utilities rather than towards proprietary solutions
> >> like FME.
> >>
> >> That said, the suggestions you are looking for are inside the document
> >> andrea suggested (here the link again http://goo.gl/TXJRS):
> >>
> >> - slides 7-9 when do use what
> >> - slides 10-13 how to optimize each single geotiff
> >> - slides 17-19 how to prepare a mosaic
> >>
> >> and so on.
> >>
> >> If you check on the web you'll also find other docs that compare tile
> >> sizes rather than types of compression, e.g.
> >> http://www.fosslc.org/drupal/content/tuning-gdal-raster-performance
> >>
> >> Regards,
> >> Simone Giannecchini
> >> ==
> >> Our support, Your Success! Visit http://opensdi.geo-solutions.it for
> >> more information.
> >> ==
> >>
> >> Ing. Simone Giannecchini
> >> @simogeo
> >> Founder/Director
> >>
> >> GeoSolutions S.A.S.
> >> Via Poggio alle Viti 1187
> >> 55054 Massarosa (LU)
> >> Italy
> >> phone: [phone]
> >> fax: [phone]
> >> mob: [phone]
> >>
> >> http://www.geo-solutions.it
> >> http://twitter.com/geosolutions _it
> >>
> >> -------------------------------------------------------
> >>
> >>
> >> On Thu, Dec 27, 2012 at 2:20 PM, Jonathan Moules
> >> <***@warwickshire.gov.uk> wrote:
> >> > Hi list,
> >> > Following this up, I've decided to go the GeoTIFF/BigTiff route with
> >> > inner
> >> > tiling and overviews based on Andrea's document.
> >> > But unfortunately there doesn't seem to be any documentation on the
> >> > GeoServer pages about this stuff. I've never used GDAL before and
> >> > generally
> >> > avoid the command line (there's a reason we're trying for GeoServer
> not
> >> > MapServer ;-) ) so am not sure where to start.
> >> >
> >> > My source input is hundreds/thousands of LZW compressed GeoTIFF tiles.
> >> > These
> >> > need to be mosaiced together (I can do that in FME easily enough).
> >> > But then what? Do I compress them at this point? Or inner tiling? Or
> >> > Overview? Does it matter what order these things are done?
> >> > And what settings should I use? For instance Andrea's document gives a
> >> > "blocksize" of 512, but Jukka's comments were on the order of 10,000
> and
> >> > the
> >> > default is 256.
> >> >
> >> > Is there a tutorial out there on this I've failed to find?
> >> > Thanks,
> >> > Jonathan
> >> >
> >> >
> >> > On 21 December 2012 14:45, Jonathan Moules
> >> > <***@warwickshire.gov.uk> wrote:
> >> >>
> >> >> Thanks Andrea, this looks like a very useful document although it is
> >> >> looking like I'm not going to be able to do this with FME from what I
> >> >> can
> >> >> see.
> >> >> Cheers,
> >> >> Jonathan
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On 20 December 2012 21:06, Andrea Aime <***@geo-solutions.it
> >
> >> >> wrote:
> >> >>>
> >> >>> On Thu, Dec 20, 2012 at 12:16 PM, Jonathan Moules
> >> >>> <***@warwickshire.gov.uk> wrote:
> >> >>>>
> >> >>>> Hi Jukka,
> >> >>>> Thanks for the information.
> >> >>>>
> >> >>>> Relating to the TIFF's, I'd still prefer to use one single tool
> >> >>>> (specifically FME) for tile/image creation. FME can create
> pyramids,
> >> >>>> but not
> >> >>>> as part of a GeoTIFF. So I can create separate TIFF files and tile
> >> >>>> them if
> >> >>>> necessary too (either internal tiling or regular tiling), but how
> do
> >> >>>> I get
> >> >>>> these into GeoServer?
> >> >>>>
> >> >>>> Is there a resource out there which describes the different Raster
> >> >>>> serving extensions and the advantages/disadvantages of each?
> >> >>>
> >> >>>
> >> >>> Here:
> >> >>>
> >> >>>
> http://demo.geo-solutions.it/share/foss4g2011/gs_steroids_sgiannec_foss4g2011.pdf
> >> >>>
> >> >>> Cheers
> >> >>> Andrea
> >> >>>
> >> >>> --
> >> >>> ==
> >> >>> Our support, Your Success! Visit http://opensdi.geo-solutions.itfor
> >> >>> more
> >> >>> information.
> >> >>> ==
> >> >>>
> >> >>> Ing. Andrea Aime
> >> >>> @geowolf
> >> >>> Technical Lead
> >> >>>
> >> >>> GeoSolutions S.A.S.
> >> >>> Via Poggio alle Viti 1187
> >> >>> 55054 Massarosa (LU)
> >> >>> Italy
> >> >>> phone: [phone]
> >> >>> fax: [phone]
> >> >>> mob: [phone]
> >> >>>
> >> >>> http://www.geo-solutions.it
> >> >>> http://twitter.com/geosolutions_it
> >> >>>
> >> >>> -------------------------------------------------------
> >> >>
> >> >>
> >> >
> >> >
> >> >
> >> > This transmission is intended for the named addressee(s) only and may
> >> > contain sensitive or protectively marked material up to RESTRICTED and
> >> > should be handled accordingly. Unless you are the named addressee (or
> >> > authorised to receive it for the addressee) you may not copy or use
> it,
> >> > or
> >> > disclose it to anyone else. If you have received this transmission in
> >> > error
> >> > please notify the sender immediately. All email traffic sent to or
> from
> >> > us,
> >> > including without limitation all GCSX traffic, may be subject to
> >> > recording
> >> > and/or monitoring in accordance with relevant legislation.
> >> >
> >> >
> >> >
> ------------------------------------------------------------------------------
> >> > Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
> >> > MVC, Windows 8 Apps, JavaScript and much more. Keep your skills
> current
> >> > with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
> >> > MVPs and experts. ON SALE this month only -- learn more at:
> >> > http://p.sf.net/sfu/learnmore_122712
> >> > ______________________
_________________________
> >> > Geoserver-users mailing list
> >> > Geoserver-***@lists.sourceforge.net
> >> > https://lists.sourceforge.net/lists/listinfo/geoserver-users
> >> >
> >
> >
> >
> >
> > This transmission is intended for the named addressee(s) only and may
> > contain sensitive or protectively marked material up to RESTRICTED and
> > should be handled accordingly. Unless you are the named addressee (or
> > authorised to receive it for the addressee) you may not copy or use it,
> or
> > disclose it to anyone else. If you have received this transmission in
> error
> > please notify the sender immediately. All email traffic sent to or from
> us,
> > including without limitation all GCSX traffic, may be subject to
> recording
> > and/or monitoring in accordance with relevant legislation.
>


This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

[Rahkonen Jukka]

Hi,

About absurdly large GDAL overview file, it is so big because you ask GDAL to create it for you as uncompressed. Gdaladdo document page has an example about how to create compressed overviews.
Page: http://www.gdal.org/gdaladdo.html
Example:

gdaladdo --config COMPRESS_OVERVIEW JPEG --config PHOTOMETRIC_OVERVIEW YCBCR
--config INTERLEAVE_OVERVIEW PIXEL rgb_dataset.ext 2 4 8 16

That's good for aerial images. For Ordnance Survey rasters I guess --config COMPRESS_OVERVIEW DEFLATE would suit best.

About number of levels, if your original has 160000 pixels start dividing it by 2:
80000, 40000, 20000, 10000, 5000, 2500, 1250, 625, 312, 156 STOP
Thus, create 10 (or 9) overview levels starting as 2 4 8...
I am not sure but I think that 1 2 4 8 ... will lead to the same result because factor 1 overview level exists and it will not be created at least as internal overviews (the original data) but you can easily test it yourself. Anyway, factor 1 overview does not make sense so you can safely start from 2 just as you were thinking.

-Jukka Rahkonen-
________________________________
Lähettäjä: Jonathan Moules [***@warwickshire.gov.uk]
Lähetetty: 2. tammikuuta 2013 18:12
Vastaanottaja: Simone Giannecchini
Kopio: geoserver-***@lists.sourceforge.net
Aihe: Re: [Geoserver-users] Rasters, Tiles, and FME

Hey Simone,

Thanks for your thorough reply. Useful!

- Resampling:
I've done some experimenting with Resampling since my email and, at least for the data I was testing, Nearest Neighbour came out best (though obviously entirely subjective). This was for a regular RGB map (Ordnance Survey MiniScale specifically). There was also a significant difference in file sizes, with Nearest Neighbour being the smallest, and Cubic being the largest (double!). I didn't expect such a significant difference, may be worth noting somewhere.

- External Overviews:
Thanks, but I'd found out how to create them with GDAL. My issue is - how do I get GeoServer to use them? Does it just pick them up automatically or do I need some sort of particular directory structure?
GDAL seems to create a single, absurdly huge file, so I guess it doesn't compress the external overviews. Based on what I'm seeing, I'm not sure if FME can create them; I expected multiple separate files (one for each layer), but the "external overviews" are basically identical to internal overviews but minus the source data.

-Overviews:
How do you determine how many levels to create? I remember this page from years ago when last I used GeoServer: http://docs.geoserver.org/stable/en/user/tutorials/imagemosaic-jdbc/imagemosaic-jdbc_tutorial.html#how-many-pyramids-are-needed - does this carry over to GeoTIFF's overviews?
I may be reading this wrong, but the example in the PDF has 7 layers but only really needs about 3 as after that they're smaller than the 1:1 tiles.

Is there any point in creating pyramids with a value of 1 (as in, 1 2 4 8...) ? GDAL lets you do this but if I'm understanding this correctly, they're entirely superfluous (I mention it because Russ's example he generously posted starts at 1).


Tile size:
Might there be a formula for calculating optimal size?
For example, if I have a 94488*157480 pixel GeoTIFF (my largest - about 2.5GB compressed), if I divide them by 512 I get
184*307 tiles = 56,488 tiles

For 256 I get:
368*614 tiles = 225,952 tiles

So if there is/was a formula, it might be easier to figure out what size to go with, or at what point you'd want to be scaling up to ImageMosaic.
Based on disk seeks etc, and some experimentation, I'd guess that a number could be contrived allowing the documentation to say:
"If you end up with > X,000 tiles, go with ImageMosiac" and "Aim for between X,000 and X0,000 tiles in a GeoTIFF"
Just a thought, would be helpful for optimisation.

Documentation:
I was considering contributing to it for this but at this point certainly don't know enough (you may have noticed from my above questions ;-) ).
I would however suggest that this section of the help: http://docs.geoserver.org/stable/en/user/production/data.html?highlight=gdaladdo#pick-the-best-performing-coverage-formats - It would gain from the info on pages 7-9 of your PDF.

Cheers!
Jonathan


On 2 January 2013 15:03, Simone Giannecchini <***@geo-solutions.it<mailto:***@geo-solutions.it>> wrote:
Ciao Jonathan,
please find my answers inline below...

Regards,
Simone Giannecchini
==
Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information.
==

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: [phone]
fax: [phone]
mob: [phone]

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------


On Wed, Jan 2, 2013 at 2:01 PM, Jonathan Moules
<***@warwickshire.gov.uk<mailto:***@warwickshire.gov.uk>> wrote:
> Hi Simone,
> I did use pages 7-9 to decide to go for BigTIFF and so 17-19 isn't
> applicable as I'm not going to ImageMosaic.
>
> While pages 10-13 are useful, its a slideshow and so lacks details which
> were probably included in the spoken part of the presentation. Its the
> details that I'm looking for. What tile size is best? What resampling
> algorithm? What overview size/scales? Etc.
> Don't get me wrong, its an excellent document, but it only provides an
> overview so can't substitute for actual detailed documentation.
>
> Your second link answers the tile-size question, but specific for MapServer
> (i.e. 256).
>
> Also, page 11 hints at "external overviews" but there's no indication as to
> how to create them or use them with GeoServer. FME can create them as I'm
> sure GDAL can too, but again, I don't know how to use External Overviews
> with Geoserver. A search of the docs for "external overviews" finds nothing
> and a google only finds that PDF.
> Does anyone have more information on how to use these with GeoServer?
>

No worries, actually this feedback gives me some hints on how to
update the presentation :)

That said, the considerations done for MapServer applies more or less
to GeoServer as they are more related
to how to geotiff works rather than to how MapServer or GeoServer works.

Anyway, my suggestions are as follows:
- tile size-
256 or for very large bigtiff 512 is my preferred size.

Using a tile size that is too small may result in too many seeks
operations on a side and on the other side might result into a TIFF
directory explosion.
The more tiles we have the more info we need to encode in the TIFF
Directory, which is essentially a list of offset locations that tells
us where the data is on disk.
For bigtiff where we have thousands of tiles it can be MB big.

- Resampling -
This depends on the data.
For RGB like data (orthos and the like) I would gor for higher order
interpolation to reduce aliasing (cripsy images).
For data like DEM or other data that contains real values to which you
will want to apply a color map I would for for Nearest neighbor to NOT
introduce artificial values.
In some case you might want to use bilinear of average but this
depends much on what you want to do.

- Overviews -
I usually use steps of 2, as many of them as possible although I
usually don't create overviews smaller than the tile size

- External Overviews -
This works only with pure geotiff (or bigtiff as well as the plugin is
the same). ImageMosaic does not (yet) support that.
External overviews are useful just in case you don't want to touch the
original files or in case they do not support inner overviews (which
is not the case for geotiff).

Creating external overviews can be done with gdaladdo, check this link
http://www.gdal.org/gdaladdo.html


> I'm fine with no FME specific documentation and wouldn't really expect
> any, but because I can't find things in the generic documentation (i.e., any
> of the above but especially external overviews), I can't necessarily
> implement an optimal solution with any utility, be it FME or GDAL.


Let me know if this helps. I will try to update those slides as well
as to create some additional docs.

You might even want to contribute some doc once you get to where you
needed to go :)

>
> Thanks,
> Jonathan
>
>
> On 2 January 2013 11:25, Simone Giannecchini
> <***@geo-solutions.it<mailto:***@geo-solutions.it>> wrote:
>>
>> Ciao Jonathan,
>> I don't know about FME therefore I cannot comment on how to do such
>> things with it. I hardly believe many people on this list (as well as
>> on other OS oriented lists) will know it much.
>> This to say that most intructions you'll find will be oriented towards
>> OS tools like GDAL utilities rather than towards proprietary solutions
>> like FME.
>>
>> That said, the suggestions you are looking for are inside the document
>> andrea suggested (here the link again http://goo.gl/TXJRS):
>>
>> - slides 7-9 when do use what
>> - slides 10-13 how to optimize each single geotiff
>> - slides 17-19 how to prepare a mosaic
>>
>> and so on.
>>
>> If you check on the web you'll also find other docs that compare tile
>> sizes rather than types of compression, e.g.
>> http://www.fosslc.org/drupal/content/tuning-gdal-raster-performance
>>
>> Regards,
>> Simone Giannecchini
>> ==
>> Our support, Your Success! Visit http://opensdi.geo-solutions.it for
>> more information.
>> ==
>>
>> Ing. Simone Giannecchini
>> @simogeo
>> Founder/Director
>>
>> GeoSolutions S.A.S.
>> Via Poggio alle Viti 1187
>> 55054 Massarosa (LU)
>> Italy
>> phone: [phone]
>> fax: [phone]
>> mob: [phone]
>>
>> http://www.geo-solutions.it
>> http://twitter.com/geosolutions _it
>>
>> -------------------------------------------------------
>>
>>
>> On Thu, Dec 27, 2012 at 2:20 PM, Jonathan Moules
>> <***@warwickshire.gov.uk<mailto:***@warwickshire.gov.uk>> wrote:
>> > Hi list,
>> > Following this up, I've decided to go the GeoTIFF/BigTiff route with
>> > inner
>> > tiling and overviews based on Andrea's document.
>> > But unfortunately there doesn't seem to be any documentation on the
>> > GeoServer pages about this stuff. I've never used GDAL before and
>> > generally
>> > avoid the command line (there's a reason we're trying for GeoServer not
>> > MapServer ;-) ) so am not sure where to start.
>> >
>> > My source input is hundreds/thousands of LZW compressed GeoTIFF tiles.
>> > These
>> > need to be mosaiced together (I can do that in FME easily enough).
>> > But then what? Do I compress them at this point? Or inner tiling? Or
>> > Overview? Does it matter what order these things are done?
>> > And what settings should I use? For instance Andrea's document gives a
>> > "blocksize" of 512, but Jukka's comments were on the order of 10,000 and
>> > the
>> > default is 256.
>> >
>> > Is there a tutorial out there on this I've failed to find?
>> > Thanks,
>> > Jonathan
>> >
>> >
>> > On 21 December 2012 14:45, Jonathan Moules
>> > <***@warwickshire.gov.uk<mailto:***@warwickshire.gov.uk>> wrote:
>> >>
>> >> Thanks Andrea, this looks like a very useful document although it is
>> >> looking like I'm not going to be able to do this with FME from what I
>> >> can
>> >> see.
>> >> Cheers,
>> >> Jonathan
>> >>
>> >>
>> >>
>> >>
>> >> On 20 December 2012 21:06, Andrea Aime <***@geo-solutions.it<mailto:***@geo-solutions.it>>
>> >> wrote:
>> >>>
>> >>> On Thu, Dec 20, 2012 at 12:16 PM, Jonathan Moules
>> >>> <***@warwickshire.gov.uk<mailto:***@warwickshire.gov.uk>> wrote:
>> >>>>
>> >>>> Hi Jukka,
>> >>>> Thanks for the information.
>> >>>>
>> >>>> Relating to the TIFF's, I'd still prefer to use one single tool
>> >>>> (specifically FME) for tile/image creation. FME can create pyramids,
>> >>>> but not
>> >>>> as part of a GeoTIFF. So I can create separate TIFF files and tile
>> >>>> them if
>> >>>> necessary too (either internal tiling or regular tiling), but how do
>> >>>> I get
>> >>>> these into GeoServer?
>> >>>>
>> >>>> Is there a resource out there which describes the different Raster
>> >>>> serving extensions and the advantages/disadvantages of each?
>> >>>
>> >>>
>> >>> Here:
>> >>>
>> >>> http://demo.geo-solutions.it/share/foss4g2011/gs_steroids_sgiannec_foss4g2011.pdf
>> >>>
>> >>> Cheers
>> >>> Andrea
>> >>>
>> >>> --
>> >>> ==
>> >>> Our support, Your Success! Visit http://opensdi.geo-solutions.it for
>> >>> more
>> >>> information.
>> >>> ==
>> >>>
>> >>> Ing. Andrea Aime
>> >>> @geowolf
>> >>> Technical Lead
>> >>>
>> >>> GeoSolutions S.A.S.
>> >>> Via Poggio alle Viti 1187
>> >>> 55054 Massarosa (LU)
>> >>> Italy
>> >>> phone: [phone]
>> >>> fax: [phone]
>> >>> mob: [phone]
>> >>>
>> >>> http://www.geo-solutions.it
>> >>> http://twitter.com/geosolutions_it
>> >>>
>> >>> -------------------------------------------------------
>> >>
>> >>
>> >
>> >
>> >
>> > This transmission is intended for the named addressee(s) only and may
>> > contain sensitive or protectively marked material up to RESTRICTED and
>> > should be handled accordingly. Unless you are the named addressee (or
>> > authorised to receive it for the addressee) you may not copy or use it,
>> > or
>> > disclose it to anyone else. If you have received this transmission in
>> > error
>> > please notify the sender immediately. All email traffic sent to or from
>> > us,
>> > including without limitation all GCSX traffic, may be subject to
>> > recording
>> > and/or monitoring in accordance with relevant legislation.
>> >
>> >
>> > ------------------------------------------------------------------------------
>> > Master Visual Studio, SharePoint, SQL, ASP.NET<http://ASP.NET>, C# 2012, HTML5, CSS,
>> > MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current
>> > with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
>> > MVPs and experts. ON SALE this month only -- learn more at:
>> > http://p.sf.net/sfu/learnmore_122712
>> > _____ _________________ _________________________
>> > Geoserver-users mailing list
>> > Geoserver-***@lists.sourceforge.net<mailto:Geoserver-***@lists.sourceforge.net>
>> > https://lists.sourceforge.net/lists/listinfo/geoserver-users
>> >
>
>
>
>
> This transmission is intended for the named addressee(s) only and may
> contain sensitive or protectively marked material up to RESTRICTED and
> should be handled accordingly. Unless you are the named addressee (or
> authorised to receive it for the addressee) you may not copy or use it, or
> disclose it to anyone else. If you have received this transmission in error
> please notify the sender immediately. All email traffic sent to or from us,
> including without limitation all GCSX traffic, may be subject to recording
> and/or monitoring in accordance with relevant legislation.



This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

[Jonathan Moules]

Hi Jukka,
Thanks for the reply. DEFLATE is definitely the best compression for
Ordnance Survey maps, much smaller than LZW.

For reference, a scale factor of "1" does create pyramids in an internal
file (I tested it, hence my querying), and as you say, they seem to be
entirely superfluous.

Jonathan


On 7 January 2013 21:03, Rahkonen Jukka <***@mmmtike.fi> wrote:

> Hi,
>
> About absurdly large GDAL overview file, it is so big because you ask GDAL
> to create it for you as uncompressed. Gdaladdo document page has an example
> about how to create compressed overviews.
> Page: http://www.gdal.org/gdaladdo.html
> Example:
>
> gdaladdo --config COMPRESS_OVERVIEW JPEG --config PHOTOMETRIC_OVERVIEW
> YCBCR
> --config INTERLEAVE_OVERVIEW PIXEL rgb_dataset.ext 2 4 8 16
>
> That's good for aerial images. For Ordnance Survey rasters I guess
> --config COMPRESS_OVERVIEW DEFLATE would suit best.
>
> About number of levels, if your original has 160000 pixels start dividing
> it by 2:
> 80000, 40000, 20000, 10000, 5000, 2500, 1250, 625, 312, 156 STOP
> Thus, create 10 (or 9) overview levels starting as 2 4 8...
> I am not sure but I think that 1 2 4 8 ... will lead to the same result
> because factor 1 overview level exists and it will not be created at least
> as internal overviews (the original data) but you can easily test it
> yourself. Anyway, factor 1 overview does not make sense so you can safely
> start from 2 just as you were thinking.
>
> -Jukka Rahkonen-
> ________________________
________
> Lähettäjä: Jonathan Moules [***@warwickshire.gov.uk]
> Lähetetty: 2. tammikuuta 2013 18:12
> Vastaanottaja: Simone Giannecchini
> Kopio: geoserver-***@lists.sourceforge.net
> Aihe: Re: [Geoserver-users] Rasters, Tiles, and FME
>
> Hey Simone,
>
> Thanks for your thorough reply. Useful!
>
> - Resampling:
> I've done some experimenting with Resampling since my email and, at least
> for the data I was testing, Nearest Neighbour came out best (though
> obviously entirely subjective). This was for a regular RGB map (Ordnance
> Survey MiniScale specifically). There was also a significant difference in
> file sizes, with Nearest Neighbour being the smallest, and Cubic being the
> largest (double!). I didn't expect such a significant difference, may be
> worth noting somewhere.
>
> - External Overviews:
> Thanks, but I'd found out how to create them with GDAL. My issue is - how
> do I get GeoServer to use them? Does it just pick them up automatically or
> do I need some sort of particular directory structure?
> GDAL seems to create a single, absurdly huge file, so I guess it doesn't
> compress the external overviews. Based on what I'm seeing, I'm not sure if
> FME can create them; I expected multiple separate files (one for each
> layer), but the "external overviews" are basically identical to internal
> overviews but minus the source data.
>
> -Overviews:
> How do you determine how many levels to create? I remember this page from
> years ago when last I used GeoServer:
> http://docs.geoserver.org/stable/en/user/tutorials/imagemosaic-jdbc/imagemosaic-jdbc_tutorial.html#how-many-pyramids-are-needed- does this carry over to GeoTIFF's overviews?
> I may be reading this wrong, but the example in the PDF has 7 layers but
> only really needs about 3 as after that they're smaller than the 1:1 tiles.
>
> Is there any point in creating pyramids with a value of 1 (as in, 1 2 4
> 8...) ? GDAL lets you do this but if I'm understanding this correctly,
> they're entirely superfluous (I mention it because Russ's example he
> generously posted starts at 1).
>
>
> Tile size:
> Might there be a formula for calculating optimal size?
> For example, if I have a 94488*157480 pixel GeoTIFF (my largest - about
> 2.5GB compressed), if I divide them by 512 I get
> 184*307 tiles = 56,488 tiles
>
> For 256 I get:
> 368*614 tiles = 225,952 tiles
>
> So if there is/was a formula, it might be easier to figure out what size
> to go with, or at what point you'd want to be scaling up to ImageMosaic.
> Based on disk seeks etc, and some experimentation, I'd guess that a number
> could be contrived allowing the documentation to say:
> "If you end up with > X,000 tiles, go with ImageMosiac" and "Aim for
> between X,000 and X0,000 tiles in a GeoTIFF"
> Just a thought, would be helpful for optimisation.
>
> Documentation:
> I was considering contributing to it for this but at this point certainly
> don't know enough (you may have noticed from my above questions ;-) ).
> I would however suggest that this section of the help:
> http://docs.geoserver.org/stable/en/user/production/data.html?highlight=gdaladdo#pick-the-best-performing-coverage-formats- It would gain from the info on pages 7-9 of your PDF.
>
> Cheers!
> Jonathan
>
>
> On 2 January 2013 15:03, Simone Giannecchini <
> ***@geo-solutions.it<mailto:
> ***@geo-solutions.it>> wrote:
> Ciao Jonathan,
> please find my answers inline below...
>
> Regards,
> Simone Giannecchini
> ==
> Our support, Your Success! Visit http://opensdi.geo-solutions.it for
> more information.
> ==
>
> Ing. Simone Giannecchini
> @simogeo
> Founder/Director
>
> GeoSolutions S.A.S.
> Via Poggio alle Viti 1187
> 55054 Massarosa (LU)
> Italy
> phone: [phone]
> fax: [phone]
> mob: [phone]
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> -------------------------------------------------------
>
>
> On Wed, Jan 2, 2013 at 2:01 PM, Jonathan Moules
> <***@warwickshire.gov.uk<mailto:
> ***@warwickshire.gov.uk>> wrote:
> > Hi Simone,
> > I did use pages 7-9 to decide to go for BigTIFF and so 17-19 isn't
> > applicable as I'm not going to ImageMosaic.
> >
> > While pages 10-13 are useful, its a slideshow and so lacks details which
> > were probably included in the spoken part of the presentation. Its the
> > details that I'm looking for. What tile size is best? What resampling
> > algorithm? What overview size/scales? Etc.
> > Don't get me wrong, its an excellent document, but it only provides an
> > overview so can't substitute for actual detailed documentation.
> >
> > Your second link answers the tile-size question, but specific for
> MapServer
> > (i.e. 256).
> >
> > Also, page 11 hints at "external overviews" but there's no indication as
> to
> > how to create them or use them with GeoServer. FME can create them as I'm
> > sure GDAL can too, but again, I don't know how to use External Overviews
> > with Geoserver. A search of the docs for "external overviews" finds
> nothing
> > and a google only finds that PDF.
> > Does anyone have more information on how to use these with GeoServer?
> >
>
> No worries, actually this feedback gives me some hints on how to
> update the presentation :)
>
> That said, the considerations done for MapServer applies more or less
> to GeoServer as they are more related
> to how to geotiff works rather than to how MapServer or GeoServer works.
>
> Anyway, my suggestions are as follows:
> - tile size-
> 256 or for very large bigtiff 512 is my preferred size.
>
> Using a tile size that is too small may result in too many seeks
> operations on a side and on the other side might result into a TIFF
> directory explosion.
> The more tiles we have the more info we need to encode in the TIFF
> Directory, which is essentially a list of offset locations that tells
> us where the data is on disk.
> For bigtiff where we have thousands of tiles it can be MB big.
>
> - Resampling -
> This depends on the data.
> For RGB like data (orthos and the like) I would gor for higher order
> interpolation to reduce aliasing (cripsy images).
> For data like DEM or other data that contains real values to which you
> will want to apply a color map I would for for Nearest neighbor to NOT
> introduce artificial values.
> In some case you might want to use bilinear of average but this
> depends much on what you want to do.
>
> - Overviews -
> I usually use steps of 2, as many of them as possible although I
> usually don't create overviews smaller than the tile size
>
> - External Overviews -
> This works only with pure geotiff (or bigtiff as well as the plugin is
> the same). ImageMosaic does not (yet) support that.
> External overviews are useful just in case you don't want to touch the
> original files or in case they do not support inner overviews (which
> is not the case for geotiff).
>
> Creating external overviews can be done with gdaladdo, check this link
> http://www.gdal.org/gdaladdo.html
>
>
> > I'm fine with no FME specific documentation and wouldn't really expect
> > any, but because I can't find things in the generic documentation (i.e.,
> any
> > of the above but especially external overviews), I can't necessarily
> > implement an optimal solution with any utility, be it FME or GDAL.
>
>
> Let me know if this helps. I will try to update those slides as well
> as to create some additional docs.
>
> You might even want to contribute some doc once you get to where you
> needed to go :)
>
> >
> > Thanks,
> > Jonathan
> >
> >
> > On 2 January 2013 11:25, Simone Giannecchini
> > <***@geo-solutions.it<mailto:
> ***@geo-solutions.it>> wrote:
> >>
> >> Ciao Jonathan,
> >> I don't know about FME therefore I cannot comment on how to do such
> >> things with it. I hardly believe many people on this list (as well as
> >> on other OS oriented lists) will know it much.
> >> This to say that most intructions you'll find will be oriented towards
> >> OS tools like GDAL utilities rather than towards proprietary solutions
> >> like FME.
> >>
> >> That said, the suggestions you are looking for are inside the document
> >> andrea suggested (here the link again http://goo.gl/TXJRS):
> >>
> >> - slides 7-9 when do use what
> >> - slides 10-13 how to optimize each single geotiff
> >> - slides 17-19 how to prepare a mosaic
> >>
> >> and so on.
> >>
> >> If you check on the web you'll also find other docs that compare tile
> >> sizes rather than types of compression, e.g.
> >> http://www.fosslc.org/drupal/content/tuning-gdal-raster-performance
> >>
> >> Regards,
> >> Simone Giannecchini
> >> ==
> >> Our support, Your Success! Visit http://opensdi.geo-solutions.it for
> >> more information.
> >> ==
> >>
> >> Ing. Simone Giannecchini
> >> @simogeo
> >> Founder/Director
> >>
> >> GeoSolutions S.A.S.
> >> Via Poggio alle Viti 1187
> >> 55054 Massarosa (LU)
> >> Italy
> >> phone: [phone]
> >> fax: [phone]
> >> mob: [phone]
> >>
> >> http://www.geo-solutions.it
> >> http://twitter.com/geosolutions _it
> >>
> >> -------------------------------------------------------
> >>
> >>
> >> On Thu, Dec 27, 2012 at 2:20 PM, Jonathan Moules
> >> <***@warwickshire.gov.uk<mailto:
> ***@warwickshire.gov.uk>> wrote:
> >> > Hi list,
> >> > Following this up, I've decided to go the GeoTIFF/BigTiff route with
> >> > inner
> >> > tiling and overviews based on Andrea's document.
> >> > But unfortunately there doesn't seem to be any documentation on the
> >> > GeoServer pages about this stuff. I've never used GDAL before and
> >> > generally
> >> > avoid the command line (there's a reason we're trying for GeoServer
> not
> >> > MapServer ;-) ) so am not sure where to start.
> >> >
> >> > My source input is hundreds/thousands of LZW compressed GeoTIFF tiles.
> >> > These
> >> > need to be mosaiced together (I can do that in FME easily enough).
> >> > But then what? Do I compress them at this point? Or inner tiling? Or
> >> > Overview? Does it matter what order these things are done?
> >> > And what settings should I use? For instance Andrea's document gives a
> >> > "blocksize" of 512, but Jukka's comments were on the order of 10,000
> and
> >> > the
> >> > default is 256.
> >> >
> >> > Is there a tutorial out there on this I've failed to find?
> >> > Thanks,
> >> > Jonathan
> >> >
> >> >
> >> > On 21 December 2012 14:45, Jonathan Moules
> >> > <***@warwickshire.gov.uk<mailto:
> ***@warwickshire.gov.uk>> wrote:
> >> >>
> >> >> Thanks Andrea, this looks like a very useful document although it is
> >> >> looking like I'm not going to be able to do this with FME from what I
> >> >> can
> >> >> see.
> >> >> Cheers,
> >> >> Jonathan
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On 20 December 2012 21:06, Andrea Aime <***@geo-solutions.it
> <mailto:***@geo-solutions.it>>
> >> >> wrote:
> >> >>>
> >> >>> On Thu, Dec 20, 2012 at 12:16 PM, Jonathan Moules
> >> >>> <***@warwickshire.gov.uk<mailto:
> ***@warwickshire.gov.uk>> wrote:
> >> >>>>
> >> >>>> Hi Jukka,
> >> >>>> Thanks for the information.
> >> >>>>
> >> >>>> Relating to the TIFF's, I'd still prefer to use one single tool
> >> >>>> (specifically FME) for tile/image creation. FME can create
> pyramids,
> >> >>>> but not
> >> >>>> as part of a GeoTIFF. So I can create separate TIFF files and tile
> >> >>>> them if
> >> >>>> necessary too (either internal tiling or regular tiling), but how
> do
> >> >>>> I get
> >> >>>> these into GeoServer?
> >> >>>>
> >> >>>> Is there a resource out there which describes the different Raster
> >> >>>> serving extensions and the advantages/disadvantages of each?
> >> >>>
> >> >>>
> >> >>> Here:
> >> >>>
> >> >>>
> http://demo.geo-solutions.it/share/foss4g2011/gs_steroids_sgiannec_foss4g2011.pdf
> >> >>>
> >> >>> Cheers
> >> >>> Andrea
> >> >>>
> >> >>> --
> >> >>> ==
> >> >>> Our support, Your Success! Visit http://opensdi.geo-solutions.itfor
> >> >>> more
> >> >>> information.
> >> >>> ==
> >> >>>
> >> >>> Ing. Andrea Aime
> >> >>> @geowolf
> >> >>> Technical Lead
> >> >>>
> >> >>> GeoSolutions S.A.S.
> >> >>> Via Poggio alle Viti 1187
> >> >>> 55054 Massarosa (LU)
> >> >>> Italy
> >> >>> phone: [phone]
> >> >>> fax: [phone]
> >> >>> mob: [phone]
> >> >>>
> >> >>> http://www.geo-solutions.it
> >> >>> http://twitter.com/geosolutions_it
> >> >>>
> >> >>> -------------------------------------------------------
> >> >>
> >> >>
> >> >
> >> >
> >> >
> >> > This transmission is intended for the named addressee(s) only and may
> >> > contain sensitive or protectively marked material up to RESTRICTED and
> >> > should be handled accordingly. Unless you are the named addressee (or
> >> > authorised to receive it for the addressee) you may not copy or use
> it,
> >> > or
> >> > disclose it to anyone else. If you have received this transmission in
> >> > error
> >> > please notify the sender immediately. All email traffic sent to or
> from
> >> > us,
> >> > including without limitation all GCSX traffic, may be subject to
> >> > recording
> >> > and/or monitoring in accordance with relevant legislation.
> >> >
> >> >
> >> >
> ------------------------------------------------------------------------------
> >> > Master Visual Studio, SharePoint, SQL, ASP.NET<http://ASP.NET>, C#
> 2012, HTML5, CSS,
> >> > MVC, Windows 8 Apps, JavaScript and much more. Keep your skills
> current
> >> > with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
> >> > MVPs and experts. ON SALE this month only -- learn more at:
> >> > http://p.sf.net/sfu/learnmore_122712
> >> > _____ _________________
_________________________
> >> > Geoserver-users mailing list
> >> > Geoserver-***@lists.sourceforge.net<mailto:
> Geoserver-***@lists.sourceforge.net>
> >> > https://lists.sourceforge.net/lists/listinfo/geoserver-users
> >> >
> >
> >
> >
> >
> > This transmission is intended for the named addressee(s) only and may
> > contain sensitive or protectively marked material up to RESTRICTED and
> > should be handled accordingly. Unless you are the named addressee (or
> > authorised to receive it for the addressee) you may not copy or use it,
> or
> > disclose it to anyone else. If you have received this transmission in
> error
> > please notify the sender immediately. All email traffic sent to or from
> us,
> > including without limitation all GCSX traffic, may be subject to
> recording
> > and/or monitoring in accordance with relevant legislation.
>
>
>
> This transmission is intended for the named addressee(s) only and may
> contain sensitive or protectively marked material up to RESTRICTED and
> should be handled accordingly. Unless you are the named addressee (or
> authorised to receive it for the addressee) you may not copy or use it, or
> disclose it to anyone else. If you have received this transmission in error
> please notify the sender immediately. All email traffic sent to or from us,
> including without limitation all GCSX traffic, may be subject to recording
> and/or monitoring in accordance with relevant legislation.
>
>
> ------------------------------------------------------------------------------
> Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
> MVC, Windows 8 Apps, JavaScript and much more. Keep your skills current
> with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
> MVPs and experts. SALE $99.99 this month only -- learn more at:
> http://p.sf.net/sfu/learnmore_122412
> ________________________
_______________________
> Geoserver-users mailing list
> Geoserver-***@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-users
>


This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.

[Simone Giannecchini]

Ciao Jonathan,
please, read my answers inline below....

Regards,
Simone Giannecchini
==
Our support, Your Success! Visit http://opensdi.geo-solutions.it for
more information.
==

Ing. Simone Giannecchini
@simogeo
Founder/Director

GeoSolutions S.A.S.
Via Poggio alle Viti 1187
55054 Massarosa (LU)
Italy
phone: [phone]
fax: [phone]
mob: [phone]

http://www.geo-solutions.it
http://twitter.com/geosolutions_it

-------------------------------------------------------


On Wed, Jan 2, 2013 at 5:12 PM, Jonathan Moules
<***@warwickshire.gov.uk> wrote:
> Hey Simone,
>
> Thanks for your thorough reply. Useful!
>
> - Resampling:
> I've done some experimenting with Resampling since my email and, at least
> for the data I was testing, Nearest Neighbour came out best (though
> obviously entirely subjective). This was for a regular RGB map (Ordnance
> Survey MiniScale specifically). There was also a significant difference in
> file sizes, with Nearest Neighbour being the smallest, and Cubic being the
> largest (double!). I didn't expect such a significant difference, may be
> worth noting somewhere.

I belive your data comprises of raster with colormap (i.e each pixel
points to an index in a colormap rather than being a real value). In
this case the only interpolation that will retain the colormap will be
nearest neighbor. If you apply higher order interpolation (i.e
bilinear and so on) that do need to mash-up somehow adiacent values
then we need to perform a color expansion which will transform the
image from 1 band to as many band as the color model has (3 for RGB
images). Hence the image size will explode per-se.

>
> - External Overviews:
> Thanks, but I'd found out how to create them with GDAL. My issue is - how do
> I get GeoServer to use them? Does it just pick them up automatically or do I
> need some sort of particular directory structure?
> GDAL seems to create a single, absurdly huge file, so I guess it doesn't
> compress the external overviews. Based on what I'm seeing, I'm not sure if
> FME can create them; I expected multiple separate files (one for each
> layer), but the "external overviews" are basically identical to internal
> overviews but minus the source data.

If you are publishing single geotiff files GeoServer should use them
automcatically, if you are using ImageMosaic, currently there is no
way to use them.

>
> -Overviews:
> How do you determine how many levels to create? I remember this page from
> years ago when last I used GeoServer:
> http://docs.geoserver.org/stable/en/user/tutorials/imagemosaic-jdbc/imagemosaic-jdbc_tutorial.html#how-many-pyramids-are-needed
> - does this carry over to GeoTIFF's overviews?
> I may be reading this wrong, but the example in the PDF has 7 layers but
> only really needs about 3 as after that they're smaller than the 1:1 tiles.

I had a cursory look at the formula and seems correct.

>
> Is there any point in creating pyramids with a value of 1 (as in, 1 2 4
> 8...) ? GDAL lets you do this but if I'm understanding this correctly,
> they're entirely superfluous (I mention it because Russ's example he
> generously posted starts at 1).

You usually do that to retile/compress/colorconvert high resolution level.

>
>
> Tile size:
> Might there be a formula for calculating optimal size?
> For example, if I have a 94488*157480 pixel GeoTIFF (my largest - about
> 2.5GB compressed), if I divide them by 512 I get
> 184*307 tiles = 56,488 tiles
>
> For 256 I get:
> 368*614 tiles = 225,952 tiles
>
> So if there is/was a formula, it might be easier to figure out what size to
> go with, or at what point you'd want to be scaling up to ImageMosaic.
> Based on disk seeks etc, and some experimentation, I'd guess that a number
> could be contrived allowing the documentation to say:
> "If you end up with > X,000 tiles, go with ImageMosiac" and "Aim for between
> X,000 and X0,000 tiles in a GeoTIFF"
> Just a thought, would be helpful for optimisation.

Well, I guess this is part of testing + experience. It may very well
depend on where the data resides (local disk, SAN, NAS) how the
filesystem has been configured, if the tiles are compressed, etc. etc.
You would need to do some testing to choose the best ones.

I believe I will update the GeoServer on steroids presentation with
some numbers for different tile sizes, compressions and so on.

>
> Documentation:
> I was considering contributing to it for this but at this point certainly
> don't know enough (you may have noticed from my above questions ;-) ).
> I would however suggest that this section of the help:
> http://docs.geoserver.org/stable/en/user/production/data.html?highlight=gdaladdo#pick-the-best-performing-coverage-formats
> - It would gain from the info on pages 7-9 of your PDF.
>
> Cheers!
> Jonathan
>
>
> On 2 January 2013 15:03, Simone Giannecchini
> <***@geo-solutions.it> wrote:
>>
>> Ciao Jonathan,
>> please find my answers inline below...
>>
>> Regards,
>> Simone Giannecchini
>> ==
>> Our support, Your Success! Visit http://opensdi.geo-solutions.it for
>> more information.
>> ==
>>
>> Ing. Simone Giannecchini
>> @simogeo
>> Founder/Director
>>
>> GeoSolutions S.A.S.
>> Via Poggio alle Viti 1187
>> 55054 Massarosa (LU)
>> Italy
>> phone: [phone]
>> fax: [phone]
>> mob: [phone]
>>
>> http://www.geo-solutions.it
>> http://twitter.com/geosolutions_it
>>
>> -------------------------------------------------------
>>
>>
>> On Wed, Jan 2, 2013 at 2:01 PM, Jonathan Moules
>> <***@warwickshire.gov.uk> wrote:
>> > Hi Simone,
>> > I did use pages 7-9 to decide to go for BigTIFF and so 17-19 isn't
>> > applicable as I'm not going to ImageMosaic.
>> >
>> > While pages 10-13 are useful, its a slideshow and so lacks details which
>> > were probably included in the spoken part of the presentation. Its the
>> > details that I'm looking for. What tile size is best? What resampling
>> > algorithm? What overview size/scales? Etc.
>> > Don't get me wrong, its an excellent document, but it only provides an
>> > overview so can't substitute for actual detailed documentation.
>> >
>> > Your second link answers the tile-size question, but specific for
>> > MapServer
>> > (i.e. 256).
>> >
>> > Also, page 11 hints at "external overviews" but there's no indication as
>> > to
>> > how to create them or use them with GeoServer. FME can create them as
>> > I'm
>> > sure GDAL can too, but again, I don't know how to use External Overviews
>> > with Geoserver. A search of the docs for "external overviews" finds
>> > nothing
>> > and a google only finds that PDF.
>> > Does anyone have more information on how to use these with GeoServer?
>> >
>>
>> No worries, actually this feedback gives me some hints on how to
>> update the presentation :)
>>
>> That said, the considerations done for MapServer applies more or less
>> to GeoServer as they are more related
>> to how to geotiff works rather than to how MapServer or GeoServer works.
>>
>> Anyway, my suggestions are as follows:
>> - tile size-
>> 256 or for very large bigtiff 512 is my preferred size.
>>
>> Using a tile size that is too small may result in too many seeks
>> operations on a side and on the other side might result into a TIFF
>> directory explosion.
>> The more tiles we have the more info we need to encode in the TIFF
>> Directory, which is essentially a list of offset locations that tells
>> us where the data is on disk.
>> For bigtiff where we have thousands of tiles it can be MB big.
>>
>> - Resampling -
>> This depends on the data.
>> For RGB like data (orthos and the like) I would gor for higher order
>> interpolation to reduce aliasing (cripsy images).
>> For data like DEM or other data that contains real values to which you
>> will want to apply a color map I would for for Nearest neighbor to NOT
>> introduce artificial values.
>> In some case you might want to use bilinear of average but this
>> depends much on what you want to do.
>>
>> - Overviews -
>> I usually use steps of 2, as many of them as possible although I
>> usually don't create overviews smaller than the tile size
>>
>> - External Overviews -
>> This works only with pure geotiff (or bigtiff as well as the plugin is
>> the same). ImageMosaic does not (yet) support that.
>> External overviews are useful just in case you don't want to touch the
>> original files or in case they do not support inner overviews (which
>> is not the case for geotiff).
>>
>> Creating external overviews can be done with gdaladdo, check this link
>> http://www.gdal.org/gdaladdo.html
>>
>>
>> > I'm fine with no FME specific documentation and wouldn't really
>> > expect
>> > any, but because I can't find things in the generic documentation (i.e.,
>> > any
>> > of the above but especially external overviews), I can't necessarily
>> > implement an optimal solution with any utility, be it FME or GDAL.
>>
>>
>> Let me know if this helps. I will try to update those slides as well
>> as to create some additional docs.
>>
>> You might even want to contribute some doc once you get to where you
>> needed to go :)
>>
>> >
>> > Thanks,
>> > Jonathan
>> >
>> >
>> > On 2 January 2013 11:25, Simone Giannecchini
>> > <***@geo-solutions.it> wrote:
>> >>
>> >> Ciao Jonathan,
>> >> I don't know about FME therefore I cannot comment on how to do such
>> >> things with it. I hardly believe many people on this list (as well as
>> >> on other OS oriented lists) will know it much.
>> >> This to say that most intructions you'll find will be oriented towards
>> >> OS tools like GDAL utilities rather than towards proprietary solutions
>> >> like FME.
>> >>
>> >> That said, the suggestions you are looking for are inside the document
>> >> andrea suggested (here the link again http://goo.gl/TXJRS):
>> >>
>> >> - slides 7-9 when do use what
>> >> - slides 10-13 how to optimize each single geotiff
>> >> - slides 17-19 how to prepare a mosaic
>> >>
>> >> and so on.
>> >>
>> >> If you check on the web you'll also find other docs that compare tile
>> >> sizes rather than types of compression, e.g.
>> >> http://www.fosslc.org/drupal/content/tuning-gdal-raster-performance
>> >>
>> >> Regards,
>> >> Simone Giannecchini
>> >> ==
>> >> Our support, Your Success! Visit http://opensdi.geo-solutions.it for
>> >> more information.
>> >> ==
>> >>
>> >> Ing. Simone Giannecchini
>> >> @simogeo
>> >> Founder/Director
>> >>
>> >> GeoSolutions S.A.S.
>> >> Via Poggio alle Viti 1187
>> >> 55054 Massarosa (LU)
>> >> Italy
>> >> phone: [phone]
>> >> fax: [phone]
>> >> mob: [phone]
>> >>
>> >> http://www.geo-solutions.it
>> >> http://twitter.com/geosolutions _it
>> >>
>> >> -------------------------------------------------------
>> >>
>> >>
>> >> On Thu, Dec 27, 2012 at 2:20 PM, Jonathan Moules
>> >> <***@warwickshire.gov.uk> wrote:
>> >> > Hi list,
>> >> > Following this up, I've decided to go the GeoTIFF/BigTiff route with
>> >> > inner
>> >> > tiling and overviews based on Andrea's document.
>> >> > But unfortunately there doesn't seem to be any documentation on the
>> >> > GeoServer pages about this stuff. I've never used GDAL before and
>> >> > generally
>> >> > avoid the command line (there's a reason we're trying for GeoServer
>> >> > not
>> >> > MapServer ;-) ) so am not sure where to start.
>> >> >
>> >> > My source input is hundreds/thousands of LZW compressed GeoTIFF
>> >> > tiles.
>> >> > These
>> >> > need to be mosaiced together (I can do that in FME easily enough).
>> >> > But then what? Do I compress them at this point? Or inner tiling? Or
>> >> > Overview? Does it matter what order these things are done?
>> >> > And what settings should I use? For instance Andrea's document gives
>> >> > a
>> >> > "blocksize" of 512, but Jukka's comments were on the order of 10,000
>> >> > and
>> >> > the
>> >> > default is 256.
>> >> >
>> >> > Is there a tutorial out there on this I've failed to find?
>> >> > Thanks,
>> >> > Jonathan
>> >> >
>> >> >
>> >> > On 21 December 2012 14:45, Jonathan Moules
>> >> > <***@warwickshire.gov.uk> wrote:
>> >> >>
>> >> >> Thanks Andrea, this looks like a very useful document although it is
>> >> >> looking like I'm not going to be able to do this with FME from what
>> >> >> I
>> >> >> can
>> >> >> see.
>> >> >> Cheers,
>> >> >> Jonathan
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> On 20 December 2012 21:06, Andrea Aime
>> >> >> <***@geo-solutions.it>
>> >> >> wrote:
>> >> >>>
>> >> >>> On Thu, Dec 20, 2012 at 12:16 PM, Jonathan Moules
>> >> >>> <***@warwickshire.gov.uk> wrote:
>> >> >>>>
>> >> >>>> Hi Jukka,
>> >> >>>> Thanks for the information.
>> >> >>>>
>> >> >>>> Relating to the TIFF's, I'd still prefer to use one single tool
>> >> >>>> (specifically FME) for tile/image creation. FME can create
>> >> >>>> pyramids,
>> >> >>>> but not
>> >> >>>> as part of a GeoTIFF. So I can create separate TIFF files and tile
>> >> >>>> them if
>> >> >>>> necessary too (either internal tiling or regular tiling), but how
>> >> >>>> do
>> >> >>>> I get
>> >> >>>> these into GeoServer?
>> >> >>>>
>> >> >>>> Is there a resource out there which describes the different Raster
>> >> >>>> serving extensions and the advantages/disadvantages of each?
>> >> >>>
>> >> >>>
>> >> >>> Here:
>> >> >>>
>> >> >>>
>> >> >>> http://demo.geo-solutions.it/share/foss4g2011/gs_steroids_sgiannec_foss4g2011.pdf
>> >> >>>
>> >> >>> Cheers
>> >> >>> Andrea
>> >> >>>
>> >> >>> --
>> >> >>> ==
>> >> >>> Our support, Your Success! Visit http://opensdi.geo-solutions.it
>> >> >>> for
>> >> >>> more
>> >> >>> information.
>> >> >>> ==
>> >> >>>
>> >> >>> Ing. Andrea Aime
>> >> >>> @geowolf
>> >> >>> Technical Lead
>> >> >>>
>> >> >>> GeoSolutions S.A.S.
>> >> >>> Via Poggio alle Viti 1187
>> >> >>> 55054 Massarosa (LU)
>> >> >>> Italy
>> >> >>> phone: [phone]
>> >> >>> fax: [phone]
>> >> >>> mob: [phone]
>> >> >>>
>> >> >>> http://www.geo-solutions.it
>> >> >>> http://twitter.com/geosolutions_it
>> >> >>>
>> >> >>> -------------------------------------------------------
>> >> >>
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > This transmission is intended for the named addressee(s) only and may
>> >> > contain sensitive or protectively marked material up to RESTRICTED
>> >> > and
>> >> > should be handled accordingly. Unless you are the named addressee (or
>> >> > authorised to receive it for the addressee) you may not copy or use
>> >> > it,
>> >> > or
>> >> > disclose it to anyone else. If you have received this transmission in
>> >> > error
>> >> > please notify the sender immediately. All email traffic sent to or
>> >> > from
>> >> > us,
>> >> > including without limitation all GCSX traffic, may be subject to
>> >> > recording
>> >> > and/or monitoring in accordance with relevant legislation.
>> >> >
>> >> >
>> >> >
>> >> > ------------------------------------------------------------------------------
>> >> > Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5, CSS,
>> >> > MVC, Windows 8 Apps, JavaScript and much more. Keep your skills
>> >> > current
>> >> > with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
>> >> > MVPs and experts. ON SALE this month only -- learn more at:
>> >> > http://p.sf.net/sfu/learnmore_122712
>> >> > _____ _________________ _________________________
>>
>> >> > Geoserver-users mailing list
>> >> > Geoserver-***@lists.sourceforge.net
>> >> > https://lists.sourceforge.net/lists/listinfo/geoserver-users
>> >> >
>> >
>> >
>> >
>> >
>> > This transmission is intended for the named addressee(s) only and may
>> > contain sensitive or protectively marked material up to RESTRICTED and
>> > should be handled accordingly. Unless you are the named addressee (or
>> > authorised to receive it for the addressee) you may not copy or use it,
>> > or
>> > disclose it to anyone else. If you have received this transmission in
>> > error
>> > please notify the sender immediately. All email traffic sent to or from
>> > us,
>> > including without limitation all GCSX traffic, may be subject to
>> > recording
>> > and/or monitoring in accordance with relevant legislation.
>
>
>
>
> This transmission is intended for the named addressee(s) only and may
> contain sensitive or protectively marked material up to RESTRICTED and
> should be handled accordingly. Unless you are the named addressee (or
> authorised to receive it for the addressee) you may not copy or use it, or
> disclose it to anyone else. If you have received this transmission in error
> please notify the sender immediately. All email traffic sent to or from us,
> including without limitation all GCSX traffic, may be subject to recording
> and/or monitoring in accordance with relevant legislation.
>
> ------------------------------------------------------------------------------
> Master Java SE, Java EE, Eclipse, Spring, Hibernate, JavaScript, jQuery
> and much more. Keep your Java skills current with LearnJavaNow -
> 200+ hours of step-by-step video tutorials by Java experts.
> SALE $49.99 this month only -- learn more at:
> http://p.sf.net/sfu/learnmore_122612
> _______________________________________________
> Geoserver-users mailing list
> Geoserver-***@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/geoserver-users
>

[Jonathan Moules]

Hi Simone,
Thanks for the information.

The best for resampling was "Average", as it produced the smoothest image
for the pyramids, at least with all of the Ordnance Survey products I've
processed.

Number of Tiles:
I think some sort of guide may be useful. I appreciate the numbers used
will have to change depending on the latency of disk access, but some broad
over-arching numbers could be created.
For example, my largest GeoTIFF has about 56,488 inner tiles (512*512
pixels) at the root layer and subjectively at least, there's a slight lag
when serving that compared to other layers which have far fewer. It's
2.87GB with pyramids, tiles and DEFLATEd, so still quite small.

Regards,
Jonathan


On 8 January 2013 08:53, Simone Giannecchini <
***@geo-solutions.it> wrote:

> Ciao Jonathan,
> please, read my answers inline below....
>
> Regards,
> Simone Giannecchini
> ==
> Our support, Your Success! Visit http://opensdi.geo-solutions.it for
> more information.
> ==
>
> Ing. Simone Giannecchini
> @simogeo
> Founder/Director
>
> GeoSolutions S.A.S.
> Via Poggio alle Viti 1187
> 55054 Massarosa (LU)
> Italy
> phone: [phone]
> fax: [phone]
> mob: [phone]
>
> http://www.geo-solutions.it
> http://twitter.com/geosolutions_it
>
> -------------------------------------------------------
>
>
> On Wed, Jan 2, 2013 at 5:12 PM, Jonathan Moules
> <***@warwickshire.gov.uk> wrote:
> > Hey Simone,
> >
> > Thanks for your thorough reply. Useful!
> >
> > - Resampling:
> > I've done some experimenting with Resampling since my email and, at least
> > for the data I was testing, Nearest Neighbour came out best (though
> > obviously entirely subjective). This was for a regular RGB map (Ordnance
> > Survey MiniScale specifically). There was also a significant difference
> in
> > file sizes, with Nearest Neighbour being the smallest, and Cubic being
> the
> > largest (double!). I didn't expect such a significant difference, may be
> > worth noting somewhere.
>
> I belive your data comprises of raster with colormap (i.e each pixel
> points to an index in a colormap rather than being a real value). In
> this case the only interpolation that will retain the colormap will be
> nearest neighbor. If you apply higher order interpolation (i.e
> bilinear and so on) that do need to mash-up somehow adiacent values
> then we need to perform a color expansion which will transform the
> image from 1 band to as many band as the color model has (3 for RGB
> images). Hence the image size will explode per-se.
>
> >
> > - External Overviews:
> > Thanks, but I'd found out how to create them with GDAL. My issue is -
> how do
> > I get GeoServer to use them? Does it just pick them up automatically or
> do I
> > need some sort of particular directory structure?
> > GDAL seems to create a single, absurdly huge file, so I guess it doesn't
> > compress the external overviews. Based on what I'm seeing, I'm not sure
> if
> > FME can create them; I expected multiple separate files (one for each
> > layer), but the "external overviews" are basically identical to internal
> > overviews but minus the source data.
>
> If you are publishing single geotiff files GeoServer should use them
> automcatically, if you are using ImageMosaic, currently there is no
> way to use them.
>
> >
> > -Overviews:
> > How do you determine how many levels to create? I remember this page from
> > years ago when last I used GeoServer:
> >
> http://docs.geoserver.org/stable/en/user/tutorials/imagemosaic-jdbc/imagemosaic-jdbc_tutorial.html#how-many-pyramids-are-needed
> > - does this carry over to GeoTIFF's overviews?
> > I may be reading this wrong, but the example in the PDF has 7 layers but
> > only really needs about 3 as after that they're smaller than the 1:1
> tiles.
>
> I had a cursory look at the formula and seems correct.
>
> >
> > Is there any point in creating pyramids with a value of 1 (as in, 1 2 4
> > 8...) ? GDAL lets you do this but if I'm understanding this correctly,
> > they're entirely superfluous (I mention it because Russ's example he
> > generously posted starts at 1).
>
> You usually do that to retile/compress/colorconvert high resolution level.
>
> >
> >
> > Tile size:
> > Might there be a formula for calculating optimal size?
> > For example, if I have a 94488*157480 pixel GeoTIFF (my largest - about
> > 2.5GB compressed), if I divide them by 512 I get
> > 184*307 tiles = 56,488 tiles
> >
> > For 256 I get:
> > 368*614 tiles = 225,952 tiles
> >
> > So if there is/was a formula, it might be easier to figure out what size
> to
> > go with, or at what point you'd want to be scaling up to ImageMosaic.
> > Based on disk seeks etc, and some experimentation, I'd guess that a
> number
> > could be contrived allowing the documentation to say:
> > "If you end up with > X,000 tiles, go with ImageMosiac" and "Aim for
> between
> > X,000 and X0,000 tiles in a GeoTIFF"
> > Just a thought, would be helpful for optimisation.
>
> Well, I guess this is part of testing + experience. It may very well
> depend on where the data resides (local disk, SAN, NAS) how the
> filesystem has been configured, if the tiles are compressed, etc. etc.
> You would need to do some testing to choose the best ones.
>
> I believe I will update the GeoServer on steroids presentation with
> some numbers for different tile sizes, compressions and so on.
>
> >
> > Documentation:
> > I was considering contributing to it for this but at this point certainly
> > don't know enough (you may have noticed from my above questions ;-) ).
> > I would however suggest that this section of the help:
> >
> http://docs.geoserver.org/stable/en/user/production/data.html?highlight=gdaladdo#pick-the-best-performing-coverage-formats
> > - It would gain from the info on pages 7-9 of your PDF.
> >
> > Cheers!
> > Jonathan
> >
> >
> > On 2 January 2013 15:03, Simone Giannecchini
> > <***@geo-solutions.it> wrote:
> >>
> >> Ciao Jonathan,
> >> please find my answers inline below...
> >>
> >> Regards,
> >> Simone Giannecchini
> >> ==
> >> Our support, Your Success! Visit http://opensdi.geo-solutions.it for
> >> more information.
> >> ==
> >>
> >> Ing. Simone Giannecchini
> >> @simogeo
> >> Founder/Director
> >>
> >> GeoSolutions S.A.S.
> >> Via Poggio alle Viti 1187
> >> 55054 Massarosa (LU)
> >> Italy
> >> phone: [phone]
> >> fax: [phone]
> >> mob: [phone]
> >>
> >> http://www.geo-solutions.it
> >> http://twitter.com/geosolutions_it
> >>
> >> -------------------------------------------------------
> >>
> >>
> >> On Wed, Jan 2, 2013 at 2:01 PM, Jonathan Moules
> >> <***@warwickshire.gov.uk> wrote:
> >> > Hi Simone,
> >> > I did use pages 7-9 to decide to go for BigTIFF and so 17-19 isn't
> >> > applicable as I'm not going to ImageMosaic.
> >> >
> >> > While pages 10-13 are useful, its a slideshow and so lacks details
> which
> >> > were probably included in the spoken part of the presentation. Its the
> >> > details that I'm looking for. What tile size is best? What resampling
> >> > algorithm? What overview size/scales? Etc.
> >> > Don't get me wrong, its an excellent document, but it only provides an
> >> > overview so can't substitute for actual detailed documentation.
> >> >
> >> > Your second link answers the tile-size question, but specific for
> >> > MapServer
> >> > (i.e. 256).
> >> >
> >> > Also, page 11 hints at "external overviews" but there's no indication
> as
> >> > to
> >> > how to create them or use them with GeoServer. FME can create them as
> >> > I'm
> >> > sure GDAL can too, but again, I don't know how to use External
> Overviews
> >> > with Geoserver. A search of the docs for "external overviews" finds
> >> > nothing
> >> > and a google only finds that PDF.
> >> > Does anyone have more information on how to use these with GeoServer?
> >> >
> >>
> >> No worries, actually this feedback gives me some hints on how to
> >> update the presentation :)
> >>
> >> That said, the considerations done for MapServer applies more or less
> >> to GeoServer as they are more related
> >> to how to geotiff works rather than to how MapServer or GeoServer works.
> >>
> >> Anyway, my suggestions are as follows:
> >> - tile size-
> >> 256 or for very large bigtiff 512 is my preferred size.
> >>
> >> Using a tile size that is too small may result in too many seeks
> >> operations on a side and on the other side might result into a TIFF
> >> directory explosion.
> >> The more tiles we have the more info we need to encode in the TIFF
> >> Directory, which is essentially a list of offset locations that tells
> >> us where the data is on disk.
> >> For bigtiff where we have thousands of tiles it can be MB big.
> >>
> >> - Resampling -
> >> This depends on the data.
> >> For RGB like data (orthos and the like) I would gor for higher order
> >> interpolation to reduce aliasing (cripsy images).
> >> For data like DEM or other data that contains real values to which you
> >> will want to apply a color map I would for for Nearest neighbor to NOT
> >> introduce artificial values.
> >> In some case you might want to use bilinear of average but this
> >> depends much on what you want to do.
> >>
> >> - Overviews -
> >> I usually use steps of 2, as many of them as possible although I
> >> usually don't create overviews smaller than the tile size
> >>
> >> - External Overviews -
> >> This works only with pure geotiff (or bigtiff as well as the plugin is
> >> the same). ImageMosaic does not (yet) support that.
> >> External overviews are useful just in case you don't want to touch the
> >> original files or in case they do not support inner overviews (which
> >> is not the case for geotiff).
> >>
> >> Creating external overviews can be done with gdaladdo, check this link
> >> http://www.gdal.org/gdaladdo.html
> >>
> >>
> >> > I'm fine with no FME specific documentation and wouldn't really
> >> > expect
> >> > any, but because I can't find things in the generic documentation
> (i.e.,
> >> > any
> >> > of the above but especially external overviews), I can't necessarily
> >> > implement an optimal solution with any utility, be it FME or GDAL.
> >>
> >>
> >> Let me know if this helps. I will try to update those slides as well
> >> as to create some additional docs.
> >>
> >> You might even want to contribute some doc once you get to where you
> >> needed to go :)
> >>
> >> >
> >> > Thanks,
> >> > Jonathan
> >> >
> >> >
> >> > On 2 January 2013 11:25, Simone Giannecchini
> >> > <***@geo-solutions.it> wrote:
> >> >>
> >> >> Ciao Jonathan,
> >> >> I don't know about FME therefore I cannot comment on how to do such
> >> >> things with it. I hardly believe many people on this list (as well as
> >> >> on other OS oriented lists) will know it much.
> >> >> This to say that most intructions you'll find will be oriented
> towards
> >> >> OS tools like GDAL utilities rather than towards proprietary
> solutions
> >> >> like FME.
> >> >>
> >> >> That said, the suggestions you are looking for are inside the
> document
> >> >> andrea suggested (here the link again http://goo.gl/TXJRS):
> >> >>
> >> >> - slides 7-9 when do use what
> >> >> - slides 10-13 how to optimize each single geotiff
> >> >> - slides 17-19 how to prepare a mosaic
> >> >>
> >> >> and so on.
> >> >>
> >> >> If you check on the web you'll also find other docs that compare tile
> >> >> sizes rather than types of compression, e.g.
> >> >> http://www.fosslc.org/drupal/content/tuning-gdal-raster-performance
> >> >>
> >> >> Regards,
> >> >> Simone Giannecchini
> >> >> ==
> >> >> Our support, Your Success! Visit http://opensdi.geo-solutions.it for
> >> >> more information.
> >> >> ==
> >> >>
> >> >> Ing. Simone Giannecchini
> >> >> @simogeo
> >> >> Founder/Director
> >> >>
> >> >> GeoSolutions S.A.S.
> >> >> Via Poggio alle Viti 1187
> >> >> 55054 Massarosa (LU)
> >> >> Italy
> >> >> phone: [phone]
> >> >> fax: [phone]
> >> >> mob: [phone]
> >> >>
> >> >> http://www.geo-solutions.it
> >> >> http://twitter.com/geosolutions _it
> >> >>
> >> >> -------------------------------------------------------
> >> >>
> >> >>
> >> >> On Thu, Dec 27, 2012 at 2:20 PM, Jonathan Moules
> >> >> <***@warwickshire.gov.uk> wrote:
> >> >> > Hi list,
> >> >> > Following this up, I've decided to go the GeoTIFF/BigTiff route
> with
> >> >> > inner
> >> >> > tiling and overviews based on Andrea's document.
> >> >> > But unfortunately there doesn't seem to be any documentation on the
> >> >> > GeoServer pages about this stuff. I've never used GDAL before and
> >> >> > generally
> >> >> > avoid the command line (there's a reason we're trying for GeoServer
> >> >> > not
> >> >> > MapServer ;-) ) so am not sure where to start.
> >> >> >
> >> >> > My source input is hundreds/thousands of LZW compressed GeoTIFF
> >> >> > tiles.
> >> >> > These
> >> >> > need to be mosaiced together (I can do that in FME easily enough).
> >> >> > But then what? Do I compress them at this point? Or inner tiling?
> Or
> >> >> > Overview? Does it matter what order these things are done?
> >> >> > And what settings should I use? For instance Andrea's document
> gives
> >> >> > a
> >> >> > "blocksize" of 512, but Jukka's comments were on the order of
> 10,000
> >> >> > and
> >> >> > the
> >> >> > default is 256.
> >> >> >
> >> >> > Is there a tutorial out there on this I've failed to find?
> >> >> > Thanks,
> >> >> > Jonathan
> >> >> >
> >> >> >
> >> >> > On 21 December 2012 14:45, Jonathan Moules
> >> >> > <***@warwickshire.gov.uk> wrote:
> >> >> >>
> >> >> >> Thanks Andrea, this looks like a very useful document although it
> is
> >> >> >> looking like I'm not going to be able to do this with FME from
> what
> >> >> >> I
> >> >> >> can
> >> >> >> see.
> >> >> >> Cheers,
> >> >> >> Jonathan
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> On 20 December 2012 21:06, Andrea Aime
> >> >> >> <***@geo-solutions.it>
> >> >> >> wrote:
> >> >> >>>
> >> >> >>> On Thu, Dec 20, 2012 at 12:16 PM, Jonathan Moules
> >> >> >>> <***@warwickshire.gov.uk> wrote:
> >> >> >>>>
> >> >> >>>> Hi Jukka,
> >> >> >>>> Thanks for the information.
> >> >> >>>>
> >> >> >>>> Relating to the TIFF's, I'd still prefer to use one single tool
> >> >> >>>> (specifically FME) for tile/image creation. FME can create
> >> >> >>>> pyramids,
> >> >> >>>> but not
> >> >> >>>> as part of a GeoTIFF. So I can create separate TIFF files and
> tile
> >> >> >>>> them if
> >> >> >>>> necessary too (either internal tiling or regular tiling), but
> how
> >> >> >>>> do
> >> >> >>>> I get
> >> >> >>>> these into GeoServer?
> >> >> >>>>
> >> >> >>>> Is there a resource out there which describes the different
> Raster
> >> >> >>>> serving extensions and the advantages/disadvantages of each?
> >> >> >>>
> >> >> >>>
> >> >> >>> Here:
> >> >> >>>
> >> >> >>>
> >> >> >>>
> http://demo.geo-solutions.it/share/foss4g2011/gs_steroids_sgiannec_foss4g2011.pdf
> >> >> >>>
> >> >> >>> Cheers
> >> >> >>> Andrea
> >> >> >>>
> >> >> >>> --
> >> >> >>> ==
> >> >> >>> Our support, Your Success! Visit http://opensdi.geo-solutions.it
> >> >> >>> for
> >> >> >>> more
> >> >> >>> information.
> >> >> >>> ==
> >> >> >>>
> >> >> >>> Ing. Andrea Aime
> >> >> >>> @geowolf
> >> >> >>> Technical Lead
> >> >> >>>
> >> >> >>> GeoSolutions S.A.S.
> >> >> >>> Via Poggio alle Viti 1187
> >> >> >>> 55054 Massarosa (LU)
> >> >> >>> Italy
> >> >> >>> phone: [phone]
> >> >> >>> fax: [phone]
> >> >> >>> mob: [phone]
> >> >> >>>
> >> >> >>> http://www.geo-solutions.it
> >> >> >>> http://twitter.com/geosolutions_it
> >> >> >>>
> >> >> >>> -------------------------------------------------------
> >> >> >>
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> > This transmission is intended for the named addressee(s) only and
> may
> >> >> > contain sensitive or protectively marked material up to RESTRICTED
> >> >> > and
> >> >> > should be handled accordingly. Unless you are the named addressee
> (or
> >> >> > authorised to receive it for the addressee) you may not copy or use
> >> >> > it,
> >> >> > or
> >> >> > disclose it to anyone else. If you have received this transmission
> in
> >> >> > error
> >> >> > please notify the sender immediately. All email traffic sent to or
> >> >> > from
> >> >> > us,
> >> >> > including without limitation all GCSX traffic, may be subject to
> >> >> > recording
> >> >> > and/or monitoring in accordance with relevant legislation.
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> ------------------------------------------------------------------------------
> >> >> > Master Visual Studio, SharePoint, SQL, ASP.NET, C# 2012, HTML5,
> CSS,
> >> >> > MVC, Windows 8 Apps, JavaScript and much more. Keep your skills
> >> >> > current
> >> >> > with LearnDevNow - 3,200 step-by-step video tutorials by Microsoft
> >> >> > MVPs and experts. ON SALE this month only -- learn more at:
> >> >> > http://p.sf.net/sfu/learnmore_122712
> >> >> > _____ ________________
_ _________________________
> >>
> >> >> > Geoserver-users mailing list
> >> >> > Geoserver-***@lists.sourceforge.net
> >> >> > https://lists.sourceforge.net/lists/listinfo/geoserver-users
> >> >> >
> >> >
> >> >
> >> >
> >> >
> >> > This transmission is intended for the named addressee(s) only and may
> >> > contain sensitive or protectively marked material up to RESTRICTED and
> >> > should be handled accordingly. Unless you are the named addressee (or
> >> > authorised to receive it for the addressee) you may not copy or use
> it,
> >> > or
> >> > disclose it to anyone else. If you have received this transmission in
> >> > error
> >> > please notify the sender immediately. All email traffic sent to or
> from
> >> > us,
> >> > including without limitation all GCSX traffic, may be subject to
> >> > recording
> >> > and/or monitoring in accordance with relevant legislation.
> >
> >
> >
> >
> > This transmission is intended for the named addressee(s) only and may
> > contain sensitive or protectively marked material up to RESTRICTED and
> > should be handled accordingly. Unless you are the named addressee (or
> > authorised to receive it for the addressee) you may not copy or use it,
> or
> > disclose it to anyone else. If you have received this transmission in
> error
> > please notify the sender immediately. All email traffic sent to or from
> us,
> > including without limitation all GCSX traffic, may be subject to
> recording
> > and/or monitoring in accordance with relevant legislation.
> >
> >
> ------------------------------------------------------------------------------
> > Master Java SE, Java EE, Eclipse, Spring, Hibernate, JavaScript, jQuery
> > and much more. Keep your Java skills current with LearnJavaNow -
> > 200+ hours of step-by-step video tutorials by Java experts.
> > SALE $49.99 this month only -- learn more at:
> > http://p.sf.net/sfu/learnmore_122612
> > _______________________________________________
> > Geoserver-users mailing list
> > Geoserver-***@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/geoserver-users
> >
>


This transmission is intended for the named addressee(s) only and may contain sensitive or protectively marked material up to RESTRICTED and should be handled accordingly. Unless you are the named addressee (or authorised to receive it for the addressee) you may not copy or use it, or disclose it to anyone else. If you have received this transmission in error please notify the sender immediately. All email traffic sent to or from us, including without limitation all GCSX traffic, may be subject to recording and/or monitoring in accordance with relevant legislation.