OutOfMemoryException loading ShapeFiles

Topics: SharpMap Project, SharpMap v0.9 / v1.x
Dec 10, 2009 at 8:51 PM

I've been loading dozens of shapefiles to our SQL Server GIS database for several months now. I can load around 20,000 shapes at a time, but the problem is that there is a memory leak somewhere that eventually throws an OutOfMemoryException in BufferedCoordinateFactory, causing me to pick up where I left off every time. Many of my shapefiles have hundreds of thousands of shapes in them, so it's quite inconvenient. I can monitor my memory usage and see it climbing the whole time. Below is my stack trace. I'm converting all the shapes from state plane projection systems to WGS84 for display in Bing.

Any ideas?

Thanks, Brad

at System.Collections.Generic.Dictionary`2.Resize()
   at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add)
   at System.Collections.Generic.Dictionary`2.set_Item(TKey key, TValue value)
   at NetTopologySuite.Coordinates.BufferedCoordinateFactory.VertexIndex.Add(DoubleComponent v0, DoubleComponent v1, DoubleComponent v2, Int32 id) in C:\Code\VisualStudio\projects\sharpmapv2\NTS\ManagedBufferedCoordinate\BufferedCoordinateFactory.cs:line 90
   at NetTopologySuite.Coordinates.BufferedCoordinateFactory.addNew(Double v0, Double v1, Double v2, Boolean is3D) in C:\Code\VisualStudio\projects\sharpmapv2\NTS\ManagedBufferedCoordinate\BufferedCoordinateFactory.cs:line 1101
   at NetTopologySuite.Coordinates.BufferedCoordinateFactory.getVertexInternal(Double x, Double y, Double z, Double w) in C:\Code\VisualStudio\projects\sharpmapv2\NTS\ManagedBufferedCoordinate\BufferedCoordinateFactory.cs:line 1030
   at NetTopologySuite.Coordinates.BufferedCoordinateFactory.getVertexInternal3D(Double x, Double y, Double z) in C:\Code\VisualStudio\projects\sharpmapv2\NTS\ManagedBufferedCoordinate\BufferedCoordinateFactory.cs:line 998
   at NetTopologySuite.Coordinates.BufferedCoordinateFactory.Create3D(Double x, Double y, Double z) in C:\Code\VisualStudio\projects\sharpmapv2\NTS\ManagedBufferedCoordinate\BufferedCoordinateFactory.cs:line 187
   at ProjNet.CoordinateSystems.Transformations.MathTransform`1.CreateCoordinate3D(Double x, Double y, Double z) in C:\Code\VisualStudio\projects\sharpmapv2\Proj.Net\ProjNet\CoordinateSystems\Transformations\MathTransform`1.cs:line 67
   at ProjNet.CoordinateSystems.Transformations.GeocentricTransform`1.metersToDegrees(TCoordinate pnt) in C:\Code\VisualStudio\projects\sharpmapv2\Proj.Net\ProjNet\CoordinateSystems\Transformations\GeocentricTransform`1.cs:line 243
   at ProjNet.CoordinateSystems.Transformations.GeocentricTransform`1.Transform(ICoordinate coordinate) in C:\Code\VisualStudio\projects\sharpmapv2\Proj.Net\ProjNet\CoordinateSystems\Transformations\GeocentricTransform`1.cs:line 303
   at ProjNet.CoordinateSystems.Transformations.ConcatenatedTransform`1.Transform(ICoordinate coordinate) in C:\Code\VisualStudio\projects\sharpmapv2\Proj.Net\ProjNet\CoordinateSystems\Transformations\ConcatenatedTransform`1.cs:line 155
   at ProjNet.CoordinateSystems.Transformations.ConcatenatedTransform`1.Transform(ICoordinate coordinate) in C:\Code\VisualStudio\projects\sharpmapv2\Proj.Net\ProjNet\CoordinateSystems\Transformations\ConcatenatedTransform`1.cs:line 155
   at SharpMap.Data.Providers.ShapeFile.ShapeFileProvider.readCoordinate() in C:\Code\VisualStudio\projects\sharpmapv2\SharpMap.Data.Providers\ShapeFileProvider\ShapeFileProvider.cs:line 1930
   at SharpMap.Data.Providers.ShapeFile.ShapeFileProvider.readSegments(Int32 lineId, Int32[] segments, ICoordinateSequence coordinates) in C:\Code\VisualStudio\projects\sharpmapv2\SharpMap.Data.Providers\ShapeFileProvider\ShapeFileProvider.cs:line 2196
   at SharpMap.Data.Providers.ShapeFile.ShapeFileProvider.readPolygon() in C:\Code\VisualStudio\projects\sharpmapv2\SharpMap.Data.Providers\ShapeFileProvider\ShapeFileProvider.cs:line 2253
   at SharpMap.Data.Providers.ShapeFile.ShapeFileProvider.readGeometry(UInt32 oid) in C:\Code\VisualStudio\projects\sharpmapv2\SharpMap.Data.Providers\ShapeFileProvider\ShapeFileProvider.cs:line 2053
   at SharpMap.Data.Providers.ShapeFile.ShapeFileProvider.getFeature(UInt32 oid, FeatureDataTable`1 table) in C:\Code\VisualStudio\projects\sharpmapv2\SharpMap.Data.Providers\ShapeFileProvider\ShapeFileProvider.cs:line 1701
   at SharpMap.Data.Providers.ShapeFile.ShapeFileProvider.GetFeatureByOid(UInt32 oid) in C:\Code\VisualStudio\projects\sharpmapv2\SharpMap.Data.Providers\ShapeFileProvider\ShapeFileProvider.cs:line 982
   at SQLGIS.LoadShapeFileToSQL(String filePath) in C:\Code\VisualStudio\projects\LoadShapeFilesToSQL08\SQLGIS.cs:line

Coordinator
Dec 10, 2009 at 8:57 PM

Hi Brad, if you use the SimpleCoordinate instead of the BufferedCoordinate you can get rid of the error.. It might run a bit quicker too..  the default GeometryServices uses the SimpleCoordinate currently.. also make sure you are runing the latest shapefile provider code.. and you may get a bit of a speedup.. cheers jd

Dec 10, 2009 at 9:50 PM
Edited Dec 14, 2009 at 9:38 PM

Sorry John, but I don't quite think I understand. My code is below. What part do you think I should change? I can basically comment out everything inside the 'for' loop including the SQL command and the problem still occurs.

 

    private void LoadShapeFileToSQL(string filePath)
    {
        InitSRIDMap();
        SqlConnection conn = new SqlConnection();
        conn.ConnectionString = @"Data Source=PE1800;Database=QA_GIS;Initial Catalog=QA_GIS;Persist Security Info=True;User ID=sa;Password=*****;Min Pool Size=10;Max Pool Size=100;Connect Timeout=100";
        SqlCommand cmd = new SqlCommand("Insert into [QA_GIS]..Parcel (MBA,Geo,ParcelNo,SeqNo,Geom,TaxYear,dtCreate) Values (@MBA,@Geo,@ParcelNo,@SeqNo,@Geometry,@TaxYear,@dtCreate)", conn);
        cmd.Parameters.Add("@MBA", SqlDbType.Decimal);
        cmd.Parameters.Add("@Geo", SqlDbType.VarChar);
        cmd.Parameters.Add("@ParcelNo", SqlDbType.VarChar);
        cmd.Parameters.Add("@SeqNo", SqlDbType.TinyInt);
        cmd.Parameters.Add("@TaxYear", SqlDbType.Int);
        cmd.Parameters.Add("@dtCreate", SqlDbType.DateTime);
        SqlParameter sp = new SqlParameter();
        sp.ParameterName = "@Geometry";
        sp.UdtTypeName = "Geometry";
        cmd.Parameters.Add(sp);
        //-----------------------------------------------------------------
        // SharpMap's Shapefile reader / writer
        ShapeFileProvider sf = null;
        GeometryServices gs = new GeometryServices();
        sf = new ShapeFileProvider(filePath + ".shp", gs.DefaultGeometryFactory, gs.CoordinateSystemFactory);
        //this keeps it from loading the entire spatial index
        sf.IsSpatiallyIndexed = false;
        sf.Open(true);

        StreamReader sr = new StreamReader(filePath + ".prj");
        String wkt = sr.ReadLine();
        ICoordinateSystem srs = gs.CoordinateSystemFactory.CreateFromWkt(wkt);
        try
        {
            IGeometryFactory gfVE = gs["EPSG:4326"];
            ICoordinateSystem wgs84 = gfVE.SpatialReference;
            ICoordinateTransformation ics = gs.CoordinateTransformationFactory.CreateFromCoordinateSystems(srs, wgs84);
            sf.CoordinateTransformation = ics;

            IGeometry geom;
            ICoordinate2D i2d;

            conn.Open();
            int seq = 1;
            int errorcnt = 0;
            int linenumber = 1;
            IFeatureDataRecord ifdr;
            uint a;
            int shapesloaded = 0;
            int rem;
            int quot;
            a = 0;
           
            for (a = 394492; a <= sf.FeatureCount; a++)
            {
                try
                {

                    ifdr = sf.GetFeatureByOid(a);

                    if (!String.IsNullOrEmpty(ifdr.GetString(ifdr.GetOrdinal("PARCELID") + 1)))
                    {

                        String prcl = ifdr.GetString(ifdr.GetOrdinal("PARCELID") + 1);
                        try
                        {
                                geom = ifdr.Geometry;                          
                                SqlBytes b = new SqlBytes(geom.AsBinary());
                                SqlGeometry g = SqlGeometry.STGeomFromWKB(b, 4326);
                                cmd.Parameters["@MBA"].Value = "1201100000";
                                cmd.Parameters["@SeqNo"].Value = seq;
                                cmd.Parameters["@Geo"].Value = prcl;
                                cmd.Parameters["@ParcelNo"].Value = prcl;
                                cmd.Parameters["@TaxYear"].Value = 2009;
                                cmd.Parameters["@dtCreate"].Value = System.DateTime.Now;
                                sp.Value = g;
                                cmd.ExecuteNonQuery();
                                seq = 1;
                                shapesloaded++;
                                quot = (int)(Math.DivRem((int)a, 100, out rem));
                                if (rem == 0) { Console.WriteLine(a.ToString()); }
                        }

                        catch (SqlException e) { Console.WriteLine("Error " + e.Message + " at ID:" + a.ToString()); errorcnt++; seq++; }
                        catch (Exception e) { Console.WriteLine("Error " + e.Message + " at ID:" + a.ToString()); errorcnt++; }
                    }
                }
                catch (Exception e) { a++; Console.WriteLine("Error " + e.Message + " at ID:" + a.ToString()); errorcnt++; }
                linenumber++;
            }

            conn.Close();

            Console.WriteLine("Total Features in ShapeFile:" + sf.FeatureCount.ToString());
            Console.WriteLine(shapesloaded.ToString() + " Shapes loaded");
            sf.Close();
            Console.WriteLine("Total Errors:" + errorcnt.ToString());
        }
        catch (Exception e) { e.ToString(); }
    }

{

 

}

private static void InitSRIDMap()SridMap.DefaultInstance = new SridMap(new[] { new SridProj4Strategy(0, new GeometryServices().CoordinateSystemFactory) });

 

Coordinator
Dec 10, 2009 at 10:25 PM

Hello Brad,

there is probably nothing wrong with your code. What john suggests is that you update
SharpMap.Utilities project and SharpMap.Providers.Shapefile project to the latest revision and perhaps add
NetTopologySuite.Coordinates.Simple to your project in order to make use of a different coordinate factory.

Hth FObermaier

Dec 10, 2009 at 11:28 PM

I updated the whole http://sharpmapv2.googlecode.com/svn/trunk, but I can’t get it to build.

From: fobermaier [mailto:notifications@codeplex.com]
Sent: Thursday, December 10, 2009 4:25 PM
To: brad.martin@accumatch.com
Subject: Re: OutOfMemoryException loading ShapeFiles [SharpMap:77767]

From: fobermaier

Hello Brad,

there is probably nothing wrong with your code. What john suggests is that you update
SharpMap.Utilities project and SharpMap.Providers.Shapefile project to the latest revision and perhaps add
NetTopologySuite.Coordinates.Simple to your project in order to make use of a different coordinate factory.

Hth FObermaier

Read the full discussion online.

To add a post to this discussion, reply to this email (SharpMap@discussions.codeplex.com)

To start a new discussion for this project, email SharpMap@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com

Coordinator
Dec 11, 2009 at 12:07 AM

Hello Brad,

there have been issues with tortoisesvn not updating externals properly.
I suggest you run an extra update on the folders NTS, GeoAPI and NPack.

Hth FObermaier

Dec 11, 2009 at 4:18 PM
Edited Dec 11, 2009 at 4:20 PM

I tried that several times. I can get NPack and GeoAPI.Net to build, but NetTopologySuite fails.

 

        IGeometry ISpatialOperator.Buffer(Double distance,

                                          BufferParameters bufferParameters )

 

Error      1881       'ISpatialOperator.Buffer' in explicit interface declaration is not a member of interface                C:\Code\VisualStudio\projects\sharpmapv2\NTS\NetTopologySuite\Geometries\Geometry.cs               1260       19                NetTopologySuite

 

 

Dec 11, 2009 at 4:27 PM

I see Coordinates.Simple project, but it's not under NetTopologySuite. I also have SharpMap.Data.Providers.ShapeFile. Same thing?

Coordinator
Dec 11, 2009 at 4:40 PM

please check subversion properties for C:\Code\VisualStudio\projects\sharpmapv2\NTS folder.

they should be:

GeoAPI https://geoapi.svn.codeplex.com/svn/branches/v2.11a
NPack https://npack.svn.codeplex.com/svn/trunk

Most likely the GeoAPI entry is wrong. After you've changed that, please update NTS folder and you should be all set.

 

bradmartin wrote:

I tried that several times. I can get NPack and GeoAPI.Net to build, but NetTopologySuite fails.

 

        IGeometry ISpatialOperator.Buffer(Double distance,

                                          BufferParameters bufferParameters )

 

Error      1881       'ISpatialOperator.Buffer' in explicit interface declaration is not a member of interface                C:\Code\VisualStudio\projects\sharpmapv2\NTS\NetTopologySuite\Geometries\Geometry.cs               1260       19                NetTopologySuite

 

 

 

Coordinator
Dec 11, 2009 at 4:44 PM
BradMartin wrote:

I see Coordinates.Simple project, but it's not under NetTopologySuite. I also have SharpMap.Data.Providers.ShapeFile. Same thing?

ManagedBufferedCoordinate hasn't been under NetTopologySuite either. Coordinates.Simple is the one John suggested.

SharpMap.Data.Providers.ShapeFile has been factored out of the main SharpMap project.

Hth

FObermaier

 

Dec 11, 2009 at 4:50 PM

I'm running v2. Are we talking about the same version?

Coordinator
Dec 11, 2009 at 4:56 PM

yes, we are talking about the same version.

Dec 11, 2009 at 6:37 PM

I checked the Subversion properties for the NTS folder like you said, but they were the same as the ones you listed. I took another update but NTS still won't build. I'm building under the .NET 2.0 framework. Could that be a problem?

Coordinator
Dec 11, 2009 at 7:53 PM

The function is defined in Geometry class (line 881 following). You could copy and paste from there.

Did you, by any chance, modify geometry class and something got messed up on merging?

 

 

Dec 14, 2009 at 7:19 PM

I'm thinking I may have better luck checking the project out again from scratch. I'm getting 70 errors to work through when I try to Rebuild the whole project. Do you recommend checking it out from  http://sharpmapv2.googlecode.com/svn/trunk or is there a better place?

Coordinator
Dec 14, 2009 at 8:36 PM

I would either choose the /trunk or branches/JDSymbolizer

If you choose /trunk, there are 2-3 web related Projects for Rendering and Presenting
with missing References to NPack. I could provide you with a patchfile if you like but
you sure can fix that yourself. Also there is a Test Project for NetTopologySuite which
will not compile. Just unload that.

If there are any pitfalls with /branches/JDSymbolizer I don't know, but I presume not
since that is the branch John has been working on recently.

Hth FObermaier

 

Dec 14, 2009 at 9:37 PM
Edited Dec 14, 2009 at 10:48 PM

I got v2 to build after checking it out again. Now I have NetTopologySuites.Coordinates.Simple added to my project. However, my code no longer runs. It's throwing an exception after I convert the geometry to binary and then try to read that binary into a SQLGeometry object. It bombs on the third line below on every record in my file. There may be a handful of invalid gemoetries, but I've already loaded almost 400,000 shapes from this file alone, so there's no way every shape is invalid. Maybe there is a more direct way to convert from Geometry class to SqlGeometry than converting to WKB?

geom = ifdr.Geometry;                           
SqlBytes b = new SqlBytes(geom.AsBinary());
SqlGeometry g = SqlGeometry.STGeomFromWKB(b, 4326);

{"24306: The Polygon input is not valid because the start and end points of the ring are not the same. Each ring of a polygon must have the same start and end points."}

I would have put this under another thread, but I still haven't gotten to the point of testing's John's solution to my original OutOfMemoryException.

Coordinator
Dec 14, 2009 at 10:24 PM
Edited Dec 14, 2009 at 10:34 PM

Could you try with BufferedCoordinate, just to see if the error is with the Coordinate factory of if its someplace else?
What is the result of ifdr.Geometry.IsValid?

As far as I know, the shapefile provider has some properties concerning (strict/lenient) its behaviour. Perhaps those work for you.

Hth FObermaier

Dec 14, 2009 at 11:30 PM

Geometry.IsValid() = true for every shape I've looked at.

I set the ShapeFile.ReadStrictNess property to Lenient. (same error)

I put a watch on the shapefile object and it's using Coordinates.Simple. Could you tell me where I should change my code above to use BufferedCoordinate?

 

This is my stack trace:
at Microsoft.SqlServer.Types.Validator.Execute(Transition transition)
   at Microsoft.SqlServer.Types.Validator.EndFigure()
   at Microsoft.SqlServer.Types.ForwardingGeoDataSink.EndFigure()
   at Microsoft.SqlServer.Types.OpenGisWkbReader.ReadLinearRing(ByteOrder byteOrder, FigureAttributes attributes, UInt32 ringNumber)
   at Microsoft.SqlServer.Types.OpenGisWkbReader.ParseWkbPolygonWithoutHeader(ByteOrder byteOrder)
   at Microsoft.SqlServer.Types.OpenGisWkbReader.ParseWkb(OpenGisType type)
   at Microsoft.SqlServer.Types.OpenGisWkbReader.ParseWkbMultiPolygonWithoutHeader(ByteOrder byteOrder)
   at Microsoft.SqlServer.Types.OpenGisWkbReader.ParseWkb(OpenGisType type)
   at Microsoft.SqlServer.Types.OpenGisWkbReader.Read(OpenGisType type, Int32 srid)
   at Microsoft.SqlServer.Types.SqlGeometry.GeometryFromBinary(OpenGisType type, SqlBytes binary, Int32 srid)
   at Microsoft.SqlServer.Types.SqlGeometry.STMPolyFromWKB(SqlBytes wkbMultiPolygon, Int32 srid)
   at SQLGIS.LoadShapeFileToSQL(String filePath) in C:\Code\VisualStudio\projects\LoadShapeFilesToSQL08\SQLGIS.cs:line 146

Coordinator
Dec 14, 2009 at 11:43 PM
Edited Dec 15, 2009 at 11:46 AM

The easiest way would be to copy GeometryServices class to e.g. GeometryServicesBC
and change every reference to Coordinate, CoordinateFactory, CoordinateSequenceFactory, ...
to the equivalent functions of the ManagedBufferedCoordinate project. If I recall correctly they
all start with BufferedCoordinate...

Then, in your code, change every call to GeometryServices to GeometryServicesBC.

Since GeometryServices Class used to use (Managed)BufferedCoordinate, you could try to
revert that class to a previous revision.

Since the simple coordinate factory was in part my work, I'm interested in a small sample
of your data, if you don't mind. Just leave me a message.

Hth FObermaier

UPDATE:

I found the error. You should be all set if you Update your local copy of GeoAPI.Net and Coordinates.Simple.

Hth FObermaier

Dec 15, 2009 at 7:02 PM

Bingo! That fixed both my invalid polygon error and the OutOfMemory exception. You rock, FObermaier!

Thanks alot for your persistence,

Brad

Dec 15, 2009 at 9:27 PM

One more thing I've noticed. Prior to this v2 update I had to adjust the FALSE NORTHING and FALSE EASTING parameters on every shapefile to get them to line up correctly in Bing. I usually had to move them south about 45 or 50 feet. Now, they appear to be lining up perfectly. I'm happy.

Coordinator
Dec 15, 2009 at 9:54 PM

Glad it works for you

FObermaier