ogr feature datatable

Topics: Data Access
Editor
Oct 7, 2013 at 5:09 PM
Hi,

I'm loading in some data from dxf into sharpmap. Works very well using the ogr extension. However when I look at the featuredataset returned from any queries the data is all in the wrong columns. It looks like if there is a null value in any column it just shifts the next bit of data into that column.

This is obviously a bit of a problem as it renders the attribute data unusable. Loading the data into QGIS (which also uses OGR) works fine, so I guess its the way the it is being converted to a datatable in sharpmap.

Is anyone else seeing this? Is there a work around?

Thanks

Rob
Coordinator
Oct 7, 2013 at 8:12 PM
mea culpa, an error introduced with changeset 91116. It has been there for over 2 years and nobody noticed.

Will fix that ASAP
Editor
Oct 7, 2013 at 8:47 PM
Ok great, let me know when its been fixed. Will the fix go into an official release, or will it only be for those that compile from source?

Whilst we are on the subject, is it correct that you cannot use the <> operator in an ogr definition query? I tried using it and wouldn't work, it seemed that only equals operators work? Is this correct?

Thanks,

Rob
Coordinator
Oct 8, 2013 at 5:48 AM
Rob, we will be releasing a new nuget package, soon
Coordinator
Oct 8, 2013 at 7:05 AM
I fixed the OgrProvider in branches/1.0, could you please verify that it works for you?

According to this document, the "<>" operator is valid in a WHERE clause and behaves exactly like "!=". These operators are not case sensitive though.

Hth FObermaier
Editor
Oct 8, 2013 at 4:21 PM
Hi,

All looks good now.

I'm getting a different error loading in a large cad file now though. It looks like ogr reads each layer in as a large wkb geometry collection, which seems to cause a 'Arithmetic operation resulted in an overflow.' error.

the error occurs in the Parse method in the GeometryFromWKB class at the line return wkb.Read(reader.BaseStream).

I've wrapped it in a try catch and once past this point loads the other layers OK.

I had brief look around and it might be to with the size of the stream buffer but I'm not sure.

Unfortunately theres no way I can upload the file for debugging because it contains sensitive information (typical!)

Thanks
Coordinator
Oct 8, 2013 at 5:34 PM
I'd assume there is sth strange with a particular geometry.
Any chance you can extract the feature in question.
Editor
Oct 8, 2013 at 6:19 PM
I'll give it a go. Its a bit tricky because it fails on the load, so i can't get the data out!

I'll try though!
Coordinator
Oct 9, 2013 at 8:32 AM
If it really is a matter of size, you may want to go through the dxf files in chunks
var p = new SharpMap.Data.Providers.OgrProvider(...);
var featureCount = p.GetFeatureCount();
const int chunkSize = 25000;
uint minFeatureId = 0;
while (minFeatureId < featureCount)
{
    p.DefinitionQuery = string.Format("FID BETWEEN({0} AND {1})", minFeatureId, minFeatureId + chunkSize-1);
    ...
    minFeatureId += chunkSize;
}
Hth FObermaier
Editor
Oct 9, 2013 at 9:00 AM
I've added a few more error traps around the code in the ogr provider.

its now crashing at this point

var byteOrder = reader.ReadByte();

it returns 3.

it makes me think I might be targeting the wrong assemblies. I'm running this on a 64-bit machine.

Would this make sense? What assemblies will I need to target? the 32-bit ones?
Editor
Oct 9, 2013 at 9:17 AM
I get error running the chunking code that you provided.

SQL Expression Parsing Error: syntax error - "FID BETWEEN (0 AND 24999)"

Might it be because DXF files to not have a FID field?
Coordinator
Oct 9, 2013 at 9:30 AM
Robert_Smart wrote:
I get error running the chunking code that you provided.

SQL Expression Parsing Error: syntax error - "FID BETWEEN (0 AND 24999)"

Might it be because DXF files to not have a FID field?
From the OGR SQL documentation
FID

Normally the feature id is a special property of a feature and not treated as an attribute of the feature. In some cases it is convenient to be able to utilize the feature id >in queries and result sets as a regular field. To do so use the name FID. The field wildcard expansions will not include the feature id, but it may be explicitly included >using a syntax like:

SELECT FID, * FROM nation
We'd have to further improve the OgrProvider to achive that
Coordinator
Oct 9, 2013 at 9:54 AM
Edited Oct 9, 2013 at 9:55 AM
Robert_Smart wrote:
I've added a few more error traps around the code in the ogr provider.

its now crashing at this point

var byteOrder = reader.ReadByte();

it returns 3.
Could you trap this condition like done below
var byteOrder = reader.ReadByte();
if (byteOrder == 3) then
System.Diagnostics.Debug.Assert(false);
end if
place a breakpoint on the Debug.Assert method and once you are there, go back up in the call stack to the ParseOgrGeometry function.
In the direct pane invoke
ogrFeature.GetGeomeryType()
and
System.Convert.ToBase64String(wkbBuffer)
Post the results here.
it makes me think I might be targeting the wrong assemblies. I'm running this on a 64-bit machine.
Would this make sense? What assemblies will I need to target? the 32-bit ones?
To use the 32bit assemblies, you need to have your project target x86 architecture instead of anycpu.
Hth FObermaier
Editor
Oct 9, 2013 at 11:15 AM
ogrGeometry.GetGeometryType() = wkbGeometryCollection


System.Convert.ToBase64String(wkbBuffer)

is over 1000 characters so will not fit in the box.

can I e-mail it to you? at what address.

I tried converting to shapefile using ogr2ogr and I think the problem might be related to the fact that the dxf layers seem to come through as geometryCollections (and very large one in this case)
Coordinator
Oct 9, 2013 at 11:46 AM
Search me on the People page and drop me a note. I'll get back to you
Editor
Oct 9, 2013 at 6:24 PM
I've changed the way i'm loading the file to loading smaller files. These load ok thanks to your fix.

however they do not overlay correctly as they all have slightly different origins. I believe there is an offset value in the head of the dxf file which will correct this.

Is there a way to access the dxf header info from sharpmap so i can apply the offset to the geometries?

thanks

Rob
Coordinator
Oct 9, 2013 at 10:10 PM
The dxf-file should be accessibe via ConnectionID property.
If you get to know the offset by parsing the header section, you need to apply add an Affine(Coordinate)Transformation. The code for doing so is somewhere in the forum and/or in the examplecodesnipplets project.

Hth FObermaier
Editor
Oct 10, 2013 at 8:57 AM
I had a look at some info online about the dxf header. It should be easy enough to parse however I think that the offsets might be held against certain block references in the file, as some of the items are in the 'correct' position and some are individual layers are offset from the correct position.

So either the block references or layers are individually offset. I would have thought that OGR might have taken this into account. If I end up having to parse the dxf file to read this info, I might as well just write a whole dxf parser that reads the whole thing!
Coordinator
Oct 10, 2013 at 10:08 AM
Robert_Smart wrote:
If I end up having to parse the dxf file to read this info, I might as well just write a whole dxf parser that reads the whole thing!
Maybe you can implement IProvider with the help of NetDxf
Editor
Oct 10, 2013 at 1:33 PM
OK I've investigated a bit more and it seems the issue may be with the way OGR interprets the Entities. I realised that the Entites were not offset, they were mirrored around the x axis of the origin.

After diving a bit deeper I have discovered that the flipped entities have the flag 230 (Z value of extrusion) set to -1. Here is an example of the DXF.

6BE9D
330
75
100
AcDbEntity
8
A-GLAZ-SILL
62
 8
100
AcDbLine
10
186199.9996695824
20
80262.49839766449
30
0.0
11
186289.9996695824
21
80262.49839766449
31
0.0
210
0.0
220
0.0
230
-1.0
0


6BE9E
330
75
100
AcDbEntity
8
A-GLAZ-SILL
62
 8
100
AcDbLine
10
186217.6776695824
20
75399.99665866671
30
0.0
11
186307.6776695824
21
75489.99665866671
31
0.0
0

OGR reads these as

6BE9D LINESTRING (-186199.99966958241 80262.498397664487, -186289.99966958241 80262.498397664487)
6BE9E LINESTRING (186217.67766958239 75399.996658666714, 186307.67766958239 75489.996658666714)

So you can see that the one with the z extrusion value of -1.0 have the x values flipped to the negative.

So the question is can ogr be configured to ignore these values? Or will I have to try and read these values manually from the dxf and then flip them back?

Thanks
Coordinator
Oct 10, 2013 at 2:47 PM
I don't see a way you can achive that, but you might better ask the expers on the gdal-dev-request mailing list or on irc #gdal

Where do the negative Z extrusion values come from in the first place?
Editor
Oct 10, 2013 at 2:59 PM
I have no idea!

This is just how they are supplied to me. I could probably use the netDXF tool that you pointed me to get the Z-value.

Thanks, for your help