[Geoserver-users] could not open .shx file/

[Imran Rajjad]

Hi list,

I get this error time to time, although it does not seem to cause any
serious harm, but I`m really curious about this. The error started
occuring after I turned on spatial indexing on a 4gb Shapefile. I`m
running GS 1.7.7 in Weblogic 10.3 on Solaris Sparc v9.



11 Jun 10:46:59 WARN [shapefile.sh] - Could not open the .shx file,
continuing assuming the .shp file is not sparse
java.io.IOException: Map failed
at org.geotools.data.shapefile.shp.IndexFile.<init>(IndexFile.java:106)
at org.geotools.data.shapefile.shp.ShapefileReader.<init>(ShapefileReader.java:162)
at org.geotools.data.shapefile.ShapefileDataStore.openShapeReader(ShapefileDataStore.java:425)
at org.geotools.data.shapefile.ShapefileRendererUtil.getShpReader(ShapefileRendererUtil.java:64)
at org.geotools.renderer.shape.ShapefileRenderer.processShapefile(ShapefileRenderer.java:542)
at org.geotools.renderer.shape.ShapefileRenderer.processStylers(ShapefileRenderer.java:376)
at org.geotools.renderer.shape.ShapefileRenderer.paint(ShapefileRenderer.java:1427)
at org.geotools.renderer.shape.ShapefileRenderer.paint(ShapefileRenderer.java:269)
at org.vfny.geoserver.wms.responses.DefaultRasterMapProducer.produceMap(DefaultRasterMapProducer.java:395)
at org.vfny.geoserver.wms.responses.map.metatile.MetatileMapProducer.produceMap(MetatileMapProducer.java:122)
at org.vfny.geoserver.wms.responses.GetMapResponse.execute(GetMapResponse.java:416)
at org.geoserver.ows.adapters.ResponseAdapter.getMimeType(ResponseAdapter.java:48)
at org.geoserver.ows.Dispatcher.response(Dispatcher.java:699)
at org.geoserver.ows.Dispatcher.handleRequestInternal(Dispatcher.java:216)
at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
at org.springframework.web.servlet


regards,
Imran

[Andrea Aime]

Post by Imran Rajjad
Hi list,
I get this error time to time, although it does not seem to cause any
serious harm, but I`m really curious about this. The error started
occuring after I turned on spatial indexing on a 4gb Shapefile. I`m
running GS 1.7.7 in Weblogic 10.3 on Solaris Sparc v9.

You are having memory mapping issues. Disable memory mapping
in the shapefile datastore configuration

Cheers
Andrea

[Imran Rajjad]

Hi,

doesnt .shx file contain index,? Do I need to turn spatial buffer off aswell?

regards
Imran

Post by Andrea Aime

Post by Imran Rajjad
Hi list,
I get this error time to time, although it does not seem to cause any
serious harm, but I`m really curious about this. The error started
occuring after I turned on spatial indexing on a 4gb Shapefile. I`m
running GS 1.7.7 in Weblogic 10.3 on Solaris Sparc v9.

You are having memory mapping issues. Disable memory mapping
in the shapefile datastore configuration
Cheers
Andrea

[Andrea Aime]

Post by Imran Rajjad
Hi,
doesnt .shx file contain index,? Do I need to turn spatial buffer off aswell?

It is not a spatial index, it map geometry number to their offset in the
.shp file. It is used mainly to support sparse files that do have holes
(normally related to deleted geometries, which is something that ArcMap
and other editors do when you remove a feature)

Cheers
Andrea

[Imran Rajjad]

I turned of memory mapping on this shape file, but the error is still
there. I turned on developer logging properties and got this trace.

11 Jun 17:55:54 DEBUG [org.geotools.data.shapefile] - Read lock:
file:/export/home/oracle/bea/geo_data/data/networkcoverage/networkcoverage.shp
by org.geotools.data.shapefile.shp.ShapefileReader
11 Jun 17:55:54 TRACE [org.geotools.data.shapefile] - Shapefile Reader
has been opened. Number open: 1
11 Jun 17:55:54 TRACE [org.geotools.data.shapefile] - IndexFile has
been opened. Number open: 1
11 Jun 17:55:54 DEBUG [org.geotools.data.shapefile] - Read lock:
file:/export/home/oracle/bea/geo_data/data/networkcoverage/networkcoverage.shx
by org.geotools.data.shapefile.shp.IndexFile
11 Jun 17:55:54 TRACE [org.geotools.data.shapefile] - Memory mapping file...
11 Jun 17:55:57 DEBUG [org.geotools.data.shapefile] - Read lock:
file:/export/home/oracle/bea/geo_data/data/networkcoverage/networkcoverage.shx
by org.geotools.data.shapefile.shp.IndexFile
11 Jun 17:55:57 WARN [org.geotools.data.shapefile.sh] - Could not open
the .shx file, continuing assuming the .shp file is not sparse
java.io.IOException: Map failed
at org.geotools.data.shapefile.shp.IndexFile.<init>(IndexFile.java:106)
at org.geotools.data.shapefile.shp.ShapefileReader.<init>(ShapefileReader.java:162)
at org.geotools.data.shapefile.ShapefileDataStore.openShapeReader(ShapefileDataStore.java:425)
at org.geotools.data.shapefile.indexed.IndexedShapefileDataStore.getAttributesReader(IndexedShapefileDataStore.java:428)
at org.geotools.data.shapefile.indexed.IndexedShapefileDataStore.getFeatureReader(IndexedShapefileDataStore.java:330)


is it still using memory mapping option?

regards,
Imran

Post by Andrea Aime

Post by Imran Rajjad
Hi,
doesnt .shx file contain index,? Do I need to turn spatial buffer off aswell?

It is not a spatial index, it map geometry number to their offset in the
.shp file. It is used mainly to support sparse files that do have holes
(normally related to deleted geometries, which is something that ArcMap
and other editors do when you remove a feature)
Cheers
Andrea

[Andrea Aime]

Post by Imran Rajjad
I turned of memory mapping on this shape file, but the error is still
there. I turned on developer logging properties and got this trace.
file:/export/home/oracle/bea/geo_data/data/networkcoverage/networkcoverage.shp
by org.geotools.data.shapefile.shp.ShapefileReader
11 Jun 17:55:54 TRACE [org.geotools.data.shapefile] - Shapefile Reader
has been opened. Number open: 1
11 Jun 17:55:54 TRACE [org.geotools.data.shapefile] - IndexFile has
been opened. Number open: 1
file:/export/home/oracle/bea/geo_data/data/networkcoverage/networkcoverage.shx
by org.geotools.data.shapefile.shp.IndexFile
11 Jun 17:55:54 TRACE [org.geotools.data.shapefile] - Memory mapping file...
file:/export/home/oracle/bea/geo_data/data/networkcoverage/networkcoverage.shx
by org.geotools.data.shapefile.shp.IndexFile
11 Jun 17:55:57 WARN [org.geotools.data.shapefile.sh] - Could not open
the .shx file, continuing assuming the .shp file is not sparse
java.io.IOException: Map failed
at org.geotools.data.shapefile.shp.IndexFile.<init>(IndexFile.java:106)
at org.geotools.data.shapefile.shp.ShapefileReader.<init>(ShapefileReader.java:162)
at org.geotools.data.shapefile.ShapefileDataStore.openShapeReader(ShapefileDataStore.java:425)
at org.geotools.data.shapefile.indexed.IndexedShapefileDataStore.getAttributesReader(IndexedShapefileDataStore.java:428)
at org.geotools.data.shapefile.indexed.IndexedShapefileDataStore.getFeatureReader(IndexedShapefileDataStore.java:330)
is it still using memory mapping option?

Look like it is

Cheers
Andrea

[Rahkonen Jukka]

Hi,

Are you sure that the shapefile is valid? You told it is 4 GB in size. 4 GB is the absolute maximum that the .shp part can reach and older ogr versions could only write .shp parts to 2 GB. I have noticed when trying to convert big data sets into shapefiles with ogr2ogr that when the .shp part reaches the maximum the process goes on and .shx and .dbf files are growing in size. However, the result is an unusable shapefile. But this is just a guess, if you can open the shapefile with some GIS program it is obviously OK.

-Jukka Rahkonen-


-----Alkuperäinen viesti-----
Lähettäjä: Andrea Aime [mailto:***@opengeo.org]
Lähetetty: pe 11.6.2010 16:11
Vastaanottaja: Imran Rajjad
Kopio: Geoserver-users
Aihe: Re: [Geoserver-users] could not open .shx file/

Post by Imran Rajjad
I turned of memory mapping on this shape file, but the error is still
there. I turned on developer logging properties and got this trace.
file:/export/home/oracle/bea/geo_data/data/networkcoverage/networkcoverage.shp
by org.geotools.data.shapefile.shp.ShapefileReader
11 Jun 17:55:54 TRACE [org.geotools.data.shapefile] - Shapefile Reader
has been opened. Number open: 1
11 Jun 17:55:54 TRACE [org.geotools.data.shapefile] - IndexFile has
been opened. Number open: 1
file:/export/home/oracle/bea/geo_data/data/networkcoverage/networkcoverage.shx
by org.geotools.data.shapefile.shp.IndexFile
11 Jun 17:55:54 TRACE [org.geotools.data.shapefile] - Memory mapping file...
file:/export/home/oracle/bea/geo_data/data/networkcoverage/networkcoverage.shx
by org.geotools.data.shapefile.shp.IndexFile
11 Jun 17:55:57 WARN [org.geotools.data.shapefile.sh] - Could not open
the .shx file, continuing assuming the .shp file is not sparse
java.io.IOException: Map failed
at org.geotools.data.shapefile.shp.IndexFile.<init>(IndexFile.java:106)
at org.geotools.data.shapefile.shp.ShapefileReader.<init>(ShapefileReader.java:162)
at org.geotools.data.shapefile.ShapefileDataStore.openShapeReader(ShapefileDataStore.java:425)
at org.geotools.data.shapefile.indexed.IndexedShapefileDataStore.getAttributesReader(IndexedShapefileDataStore.java:428)
at org.geotools.data.shapefile.indexed.IndexedShapefileDataStore.getFeatureReader(IndexedShapefileDataStore.java:330)
is it still using memory mapping option?

Look like it is

Cheers
Andrea

[Imran Rajjad]

Hi Jukka.

The shapefile is actualy a GDB file exported to shapefile. Its got
near about 100,000 parcels in it. Now since we cannot upload a gdb
file to geoserver , there are very few options , I tried puting this
file in SDO_GEOMETRY but it takes really long to respond to both wms
and wfs request. anyway I did turn off the mapping, and its still
usnig memory mapping. I guess I`ll try restarting the machine or the
process. However the error seems to be very harmless and un-noticed at
client end. When zoomed in at street level, the performance is just
fine and all the parcels are coming in nicely.

regards,
Imran

On Fri, Jun 11, 2010 at 7:21 PM, Rahkonen Jukka

Post by Rahkonen Jukka
Hi,
Are you sure that the shapefile is valid? You told it is 4 GB in size. 4 GB is the absolute maximum that the .shp part can reach and older ogr versions could only write .shp parts to 2 GB. I have noticed when trying to convert big data sets into shapefiles with ogr2ogr that when the .shp part reaches the maximum the process goes on and .shx and .dbf files are growing in size. However, the result is an unusable shapefile. But this is just a guess, if you can open the shapefile with some GIS program it is obviously OK.
-Jukka Rahkonen-
-----Alkuperäinen viesti-----
Lähetetty: pe 11.6.2010 16:11
Vastaanottaja: Imran Rajjad
Kopio: Geoserver-users
Aihe: Re: [Geoserver-users] could not open .shx file/

Post by Imran Rajjad
I turned of memory mapping on this shape file, but the error is still
there. I turned on developer logging properties and got this trace.
file:/export/home/oracle/bea/geo_data/data/networkcoverage/networkcoverage.shp
by org.geotools.data.shapefile.shp.ShapefileReader
11 Jun 17:55:54 TRACE [org.geotools.data.shapefile] - Shapefile Reader
has been opened. Number open: 1
11 Jun 17:55:54 TRACE [org.geotools.data.shapefile] - IndexFile has
been opened. Number open: 1
file:/export/home/oracle/bea/geo_data/data/networkcoverage/networkcoverage.shx
by org.geotools.data.shapefile.shp.IndexFile
11 Jun 17:55:54 TRACE [org.geotools.data.shapefile] - Memory mapping file...
file:/export/home/oracle/bea/geo_data/data/networkcoverage/networkcoverage.shx
by org.geotools.data.shapefile.shp.IndexFile
11 Jun 17:55:57 WARN [org.geotools.data.shapefile.sh] - Could not open
the .shx file, continuing assuming the .shp file is not sparse
java.io.IOException: Map failed
        at org.geotools.data.shapefile.shp.IndexFile.<init>(IndexFile.java:106)
        at org.geotools.data.shapefile.shp.ShapefileReader.<init>(ShapefileReader.java:162)
        at org.geotools.data.shapefile.ShapefileDataStore.openShapeReader(ShapefileDataStore.java:425)
        at org.geotools.data.shapefile.indexed.IndexedShapefileDataStore.getAttributesReader(IndexedShapefileDataStore.java:428)
        at org.geotools.data.shapefile.indexed.IndexedShapefileDataStore.getFeatureReader(IndexedShapefileDataStore.java:330)
is it still using memory mapping option?

Look like it is
Cheers
Andrea
------------------------------------------------------------------------------
ThinkGeek and WIRED's GeekDad team up for the Ultimate
GeekDad Father's Day Giveaway. ONE MASSIVE PRIZE to the
http://p.sf.net/sfu/thinkgeek-promo
_______________________________________________
Geoserver-users mailing list
https://lists.sourceforge.net/lists/listinfo/geoserver-users