Questions (confirmations) on rendering

Topics: SharpMap v0.9 / v1.x, WinForms Controls
Oct 10, 2011 at 9:45 AM

Hello

I've some questions or confirmations on rendering, the context is to use huge files ( more than 30 000 objects per layer), and the goal is to speed up.

1- there is no difference between pictureBox1.image=mymap.getmap() and mymap.RenderMap(myGraphic) on a speed point of view.

2- there is no way to render only one layer and keep on the other. The case is when I have a small layer over a big one, for example a labelLayer with 3 labels over a layer with 25000 geometries. The both layers are redrawn, this takes few seconds.To demonstrate that, I set a labelLayer with about 10 labels over a 36000 geom layer, it takes about 5 seconds to set the labelayer visible on or off. The idea should be to keep in memory the .net graphic of the big layer and only redraw the labels on it. Of course this doesn't seed up anything when zoomin and panning.

3- I 've compare the render speed (during loading the layer, zooming, paning,...) between shp and SQLServer 2008 (the same 'map' with 35 000 geom, I've tried with sharpMap, ImageMap and ImageBox, the results are mainly the same). The SQL is 2 to 5 times slower. I think the problem comes from that we need conversion  with SQLServer format while shp is more direct. Is it true ? Is it a way to have a sort of 'vector cache', in this case we convert only the geom which are not already converted ?

4- Maybe this an another solution to speed up SQLserver rendering ?

5- In the same way than point 2, on rendering, sharpMap render all the layers even if the layers under are masked by the upper one. Is there a way to manage this point without control the visibility property of the layer ?

Thanks in advance

regards

Eric

Coordinator
Oct 10, 2011 at 12:36 PM
  1. I would assume it only would make a difference if the size defined in Map.Size differs from myGraphics size.
  2. Currently not in the trunk. There is a branch (branches/0.9.5-Deltashell) that renders each layer in a seperate image.
    Maybe you can add the Labels to the VariableLayerCollection and just invalidate its view.
  3. The SQLServer2008 needs itself needs to convert the geometries to WKB so SharpMap can parse and process them. The "ReadCoordinates" function in SharpMap.Converters.WellKnownBinary.GeometryFromWKB is not optimized for speed.
    The ShapeFile provider has a cache, I don't know how it works though. Another approach would be a caching proxy like the one for the GDALRasterLayer, but for vectors.
  4. You may try this to see if it helps:

        private static Point[] ReadCoordinates(BinaryReader reader, WkbByteOrder byteOrder)
        {
            return ReadCoordinates(reader, byteOrder, 2);
        }

        private static Point[] ReadCoordinates(BinaryReader reader, WkbByteOrder byteOrder, int numberOfOrdinates)
        {
            // Get the number of points in this linestring.
            int numPoints = (int) ReadUInt32(reader, byteOrder);

            // Create a byte array buffer
            var bytes = new byte[numPoints*numberOfOrdinates*8];
            reader.Read(bytes, 0, bytes.Length);

            // Create a new array of coordinates.
            var coords = new Point[numPoints];

            var index = 0;
            var ordinates = new double[numberOfOrdinates];
            var offset = numberOfOrdinates*8;

            switch (byteOrder)
            {
                case WkbByteOrder.Ndr:
                    // Loop on the number of points in the ring.
                    for (int i = 0; i < numPoints; i++)
                    {
                        //copy the data
                        Buffer.BlockCopy(bytes, i*offset, ordinates, 0, offset);

                        // Add the coordinate.
                        coords[i] = new Point(ordinates[0], ordinates[1]);
                    }
                    break;

                case WkbByteOrder.Xdr:
                    // Loop on the number of points in the ring.
                    for (int i = 0; i < numPoints; i++)
                    {
                        //copy the data
                        Buffer.BlockCopy(bytes, i*offset, ordinates, 0, offset);

                        // Add the coordinate.
                        coords[i] = new Point(ReverseDouble(ordinates[0]), ReverseDouble(ordinates[1]));
                    }
                    break;
                default:
                    throw new ArgumentOutOfRangeException("byteOrder");
            }
            return coords;
        }

        private static double ReverseDouble(double val)
        {
            byte[] bytes = BitConverter.GetBytes(val);
            Array.Reverse(bytes);
            return BitConverter.ToDouble(bytes, 0);
        }

5.)

Hth FObermaier

Oct 10, 2011 at 12:46 PM

Great !

I'll try asap and keep you inform

Eric

Oct 12, 2011 at 5:31 AM

Hello,

I've try to speed up and follow yours inputs without real success. After analysis, The WKB conversion is not so bad. The slowest function, in fact, is the intersect query with the BBox use to retreive only the datas incluted in the BBox. The idea is good, we take only the geoms which were displayed, but with SQL Server it takes more time to select a part of geoms that load them all. I've tried to use Filter instead of STIntersect, to adjust the spatial index but without any real improvement. Finaly I bypass the BBox, cleaning a little bit the code and obtains good result: about 2.5 seconds to load, convert and display 30 000 objects. For comparison I got a litlle bit more than 2 sec with QGIS ( same objects but in shp) and less than 1 sec with MapInfo (in native format .tab). So it's prety good, we can achieve the same perf than QGIS.

I begin to code a cache, it's just  putting the features collection outside the method GetFeatureInView and recall it without reloaded from SQLServer each time we have to redraw. It's working for polygons, I need to extend to other geometries.

If anybody has some input to implement the cache or to have a very fast intersect query in SQL server, I 'm interested.

Regards

Eric

Coordinator
Oct 12, 2011 at 6:56 AM
ericma62 wrote:

... After analysis, The WKB conversion is not so bad. The slowest function, in fact, is the intersect query with the BBox use to retreive only the datas incluted in the BBox. The idea is good, we take only the geoms which were displayed, but with SQL Server it takes more time to select a part of geoms that load them all.

Just curious, you do have a spatial index on your table? The geometries are fairly complex (lots of vertices)? If so, you might get better performance if you shortcut just comparing intersection on bounding boxes.

Hth FObermaier

Oct 12, 2011 at 8:43 AM

Yes there is a spatial index on the table. Remarks if there is no spatial index, the method GetExtents (in Sqlserver2008.cs) take a very long time ( about 4 sec) due to the 'union' of all geom to get the layer extends.

The geometries are percels so mainly rectangles, I've try the intersect on bboxes  with .STEnvelope and got the same result: about 3-4 sec for the intersect (whatever the method used) verus 2.5 sec for loading, calculating and showing the polygons. I think it is due the SQLServer itself, I run the querie in SQL server Management Studio and it takes a long time... Maybe it's possible to improve with optimizing the spatial index but I don't see how.

Eric

Coordinator
Oct 12, 2011 at 12:25 PM

Are you sure SqlServer is using the index for the query? Check the ExecutionPlan in SqlServer Management Studio.

If not, you could provide an index hint (SELECT ... FROM "table" WITH (INDEX("NameOfYourIndex"); )

Hth FObermaier

Coordinator
Oct 12, 2011 at 12:28 PM

Hi, about 

" the method GetExtents (in Sqlserver2008.cs) take a very long time ( about 4 sec)"...

There is a constructor overload for the SQLServer layer that tells it to read the Extents from the Spatial Index instead of from doing union of all features. This speedup is huge on layers with many features that does have a spatial index.

Coordinator
Oct 12, 2011 at 12:34 PM
Edited Oct 12, 2011 at 12:41 PM

In Regard to your Caching solution.

Please have a look at the GdalRasterLayerCachingProxy. It basically stores the raster image from the last RenderLayer function along with some additional information.
When the RenderLayer function is called the next time it first checks whether bounds or size have changed. If not, the last image is blitted on the graphics context, otherwise the underlying GdalRasterLayer is queried for a new image, which then is cached and blitted on the graphics context.

You/we can do the same thing for a [Vector|Label]Layer:

  • make Style and theme implement IEquatable or INotifyPropertyChanged
  • Create a [Vector|LabelLayer]CachingProxy that either handles PropertyChanged events or extend the checks for to requery to evaluate style and theme

Hth FObermaier

Oct 14, 2011 at 9:37 AM
Edited Oct 14, 2011 at 9:39 AM

Hello

Here after some feedback.

The intersect query takes a long time when the BBox is large, It's the case when the map displayed is the entire layer. Each time the End User zoom or pan, SharpMap run an ExecuteIntersectonQuery or an GetGeometryInView (depends on if there is a theme or not) which takes about 4 sec (3 sec for the query in SQL server, 1 sec  for WKB conversion), if we add 1 sec for displaying the geoms, that gives about 5 sec. That too long for a End User to wait 4, 5 or 6 sec each time he zooms or Pans. The use of spatial index doesn't speed up anything, It's due to that the BBox is very large and intersects with 50 , 70 , 90, or 100% of the geoms, so the spatial index is not usefull. It's not the case when the zoom is 'fine', when the bbox is narrow, it intersects with few geoms ( less than 10 %) and the ExecuteIntersectonQuery is very fast.

To avoid this problem, I apply the ExecuteIntersectonQuery only if the  bbox is smaller  than layer extents/5 anf if the GetFeatureCount is under 5000. If not I memorise the datas for the whole layer and don't query SQL Server. That save the query and the WKB conversion times.

With these mods the zooming and panning times are always under 2 sec.

I have to implement the 'Image catching', used when there no pan or zoom, but just a redraw due to a change in the visibilty of a layer or adding labels,... If I have the time, I will try  next week with your inputs.

Regards

Eric

Nov 30, 2011 at 4:20 AM

i have the same problem..since BBox is very large it takes time to intersect..how to check it? can you please share the code? i need to improve the performance..please