An open letter...

Topics: SharpMap v2.0
Dec 14, 2007 at 3:15 AM
Edited Dec 14, 2007 at 3:18 AM
Today has been a long and frustrating day, so I probably should not be writing this. But, here goes. I've assembled a list of what I expect from this effort and I present it here. It is not my intent to offend, or to hijack the direction of the project. But, as a business man, I'm taking a risk in moving to an environment over which I have no control (via a checkbook). Please do me the honor of letting me know if I can expect what I have written below, or if I need to look for other solutions. And, my comments here are related to SharpMap, ProjNet, GeoAPI, and NTS. I know they are separate projects with separate forums, but clearly the driving leaders all reside here.

1) I expect there to be a set of interfaces that support the Open GIS concepts (simple feature geometry objects and coordinate transformations) and data formats (WKB, WKT, and GML). I expect to be able to mix and mingle third party components that implement these interfaces. They will become the only GIS interfaces that I will expose outside of our GIS engine.

2) I expect to be able to replace projnet with any other projection engine, if I'm willing to make sure that projection engine implements the GeoAPI coordinate system interfaces. SharpMap and NTS can have NO dependency on ProjNet or its embedded interfaces (currently the NTS project references ProjNet, but it is a supurflous reference as the project compiles if you crop it.

3) I expect to be able to replace NTS or parts of NTS with any other implementation whose components implement the GeoAPI interfaces. We have already written wrappers around some commercial third-party components, and our wrappers implement the GeoAPI interfaces. SharpMap needs to be able to operate with my alternative geometry objects as well as it does with the NTS objects (after it gets ported to NTS).

4) NTS should have all file-based IO classes removed. Several of them don't work... they even warn you not to use them. NTS should be a geometry engine only. It needs to support serialization to WKB, WKT, GML, and maybe KML via streams, but it should not support serialization at any higher level than that.

5) The SharpMap project should be independent of any data providers. It is perfectly acceptable to define a set of interfaces that must be implemented by any plug-in layers and drivers, but the data providers must live in supporting projects. That's the way 2.0 is going, right?

6) There should be an encoders project. Encoders are classses that can read or write things like shapefiles. People that need support for shapefiles in NTS should have their client applications instantiate the encoder/provider that they need, and link those to NTS via a standard recordset navigation interface... movefirst, move next, etc... I suppose the more modern dataset interface would be ok, too, as long as it doesn't require 300,000 records from a shapefile to be read into a dataset up front. We have already lifted some of the shapefile encoding stuff out of NTS and simplified it and made it work faster. We'll be glad to contribute this back... but into a separate project.

7) SharpMap should separate the rendering options into different projects. And that's happening too, right?

8) The core SharpMap project should have no "utility" UI components. That project needs to be a bare-bones, screaming gis rendering solution. If we want to share utilities like legend displays, theme editors, scale bars, etc, etc... that would be great... but put them in a separate project, please.

9) Non-Graphical GeoProcessing should be done in a separate project from SharpMap. And, here is where it gets tough. An exmple would be a function that takes two "sharpmap" polygon layers and intersects them to produce a new layer that is every permutation and combination of polygon intersections, and whose fields are the appropriate aggregation of the values from the intersecting polygons. Or a layer that assigns values from a layer of containing polygons to a layer of contained sites. Or a engine that performs surfacing or krieging on a layer of sites. At this point, I'm not sure that we have a set of interfaces that are designed for non-graphical functions to iterate over a "layer" or a "datasource" or a "dataprovider" so that these kinds of geoprocessing operations can be performed. In a perfect world, I would like the data access and navigation interfaces for layers and data sources to be defined within GeoAPI. Then it would be possible to run geoprocessing functionality without a reference to SharpMap if no presentation capabilities are required in the application. As an alternative, we could have a geoprocessing project that defines the data access interfaces necessary for layer/recordset navigation during geoprocessing, and the external data providers could just implement the additional interfaces if they wanted to be able to participate in geoprocessing.

10) And this last one is maybe the biggest concern... I was notified this afternoon that we just lost a large conctract... one where we were proposing SharpMap, et al, as the GIS solution. We have written a lot of code that works with GeoAPI and ProjNet and NTS. But we can't use any of it with SharpMap at this stage. We lost the contract because for two months, I have not been able to produce a sample application with SharpMap. I'm using the latest stable build of ProjNet, GeoAPI, and NTS with great success (the build where ProjNet interfaces were moved into GeoAPI). But I can't draw a map with any of our work. The kicker was that I could say "we'll there is some really good work going on with refactoring GeoAPI and NTS to support generics". Those are nice buzzwords for management, but in the end, they want to see a picture. In the abscence of a picture, I could not point to a document that laid out a clear reason for the refactoring, an estimate of the effort, or a timeline for completion. I couldn't point to a discussioin where a decision was made that the refacatoring was worth the effort and delay. I couldn't even point to a really solid discussion about the decision to introduce a dependency on .Net Framework 3.5 (yeah... I know... we only need 2.1, but you have to install 3.5 to get it, and that doesn't make corporate IT guys too happy). What I could point to are a series of posts that say "next week", followed by "i'm not being given the time I need by my bosses". And when it is all said an done... the people I was trying to sell on this effort said "yep... that's the problem with open source", and turned around and paid tens of thousands of dollars for commercial software liceneses and other programmers to implement their needed solutions... and the solutions were ooohhhh so simple.

Soooo I'm getting scared. We can't access the current builds through SVN because of some glitch. We continue to have the confusion of a 2.0 branch on CodePlex, but the code is really in google. And personally, I'm in the dark about why we are in this black hole of generics refactoring. I've seen enough of the developer to have little doubt that it is a very elegant enhancement... but gosh... once the interfaces from ProjNet were refactored into GeoAPI, I was sure a happy camper :) Our geoprocessing stuff with those builds is working great.

And again.... I'm really not wanting to complain. I'm new to open source. And I'm new to this project (since last june or so). Maybe this is how stuff works? But what I'm seeing here is a one man show (by a great man, fortunately), but one man. And one man shows really scare corporate guys.

So, help me out? Are my expectations in line with others involved in the project? And if so, are we in a unusual blip, or is this the way open source works?

And, I apologize in advance if I have offended anyone.
Developer
Dec 15, 2007 at 1:09 PM
1) I expect there to be a set of interfaces that support the Open GIS concepts
This is the main purpose of GeoAPI, but this project is born around NTS, so I'm expecting a lot of work to create a generic library for Geospatial definitions (like java's GeoAPI), work that I'm unable to do at the moment and that requirea lot of coordination between projects.
2) I expect to be able to replace projnet with any other projection engine, i
This could be simple when all references to concrete implementations will be removed (like in NTS, that could use stuff like DI frameworks to remove all concrete references.
3) I expect to be able to replace NTS or parts of NTS with any other implementation whose components implement the GeoAPI interfaces.
same as 2.
4) NTS should have all file-based IO classes removed. Several of them don't work... they even warn you not to use them. NTS should be a geometry engine only. It needs to support serialization to WKB, WKT, GML, and maybe KML via streams, but it should not support serialization at any higher level than that.
Kudos for you, this is my final goal, but a little bit of history... NTS was born in the past as a port of JTS, but looks not useful without any IO helper, so I've retrieved old code from GeoTools.NET (a dead project) to create basical IO operations. It's hard to me at this time to support NTS topologycal operations, is even more hard to support IO operations, howewer if you send me the bugs you have found, i could try to do my best... it's too simple to write stuff like "NTS IO sucks"... try first to use issue tracker on google code :)
10) We lost the contract because for two months, I have not been able to produce a sample application with SharpMap. I'm using the latest stable build of ProjNet, GeoAPI, and NTS with great success (the build where ProjNet interfaces were moved into GeoAPI). But I can't draw a map with any of our work.
I think that you couldn't sell a 2.0beta1 solution: simply v0.9 works well and draw great pictures (take a look at http://www.ultramap.cl/). Rory and sharpmap team is made a great work for 2.0, but this work needs time
Dec 15, 2007 at 7:00 PM
Clarification -- I didn't say "NTS IO sucks". I said it didn't belong in the same assembly as the NTS geometry operations. I understand they got implemented as a way to get testing off the ground... gotta get and save data to exercise the geometry functions. But I'm glad to hear that there is a plan to separate them out. We are doing so now, on our own. We'll post that work when we get it complete. We had already gone in and finished the implementation for the data types other than double. Stuff like that.

Thanks, sir.
Coordinator
Dec 19, 2007 at 5:59 PM
@Magnum4610:

I'm truly sorry to hear about the contract.

Although I'm fairly new to contributing to open source, I've been a user for years. Just let me be up front right away, however: open source is risky.

I've had to rely on open source many times in the past, and there are times where it has come through, and other times where it has not. In general, the more stable a project is, the less risky it is. I've tried to make it clear that v2.0 is not stable, not at all.

The situation with the software not being adequate for your needs is a very familiar one with open source. That is both a good and bad thing: you can make it what you want, but that involves more risk: missed deadlines, inadequate features, more difficult administration and support, etc. The trade off is cash. If you want to mitigate risk, you have to pay a vendor for it. It really can't be any other way - open source has to be paid for by someone: someone whose goals are similar, but very unlikely exactly like yours. In my case, we need the mapping library for our own product. When SharpMap v2.0 is adequate to meet those needs, my employer will stop paying for it, as it should. However, the code is now out there, waiting for more investment, unlike an unsupported vendor product (MapObjects springs to mind). All of the other contributors are in the same position: they work on it to the extent that they can put food on the table.

Your experience, as far as I can see, results from having to depend on code which isn't complete. In your case, I can argue that you may have been in the same position even if SharpMap development were fully funded by a responsible vendor. How many software companies can you think of who haven't announced a product, and then delayed it? If I were relying on a release of Windows or OS X or for a Google product to get a certain feature, I'd be in very, very risky territory... and these are companies who have more money than whole countries of people!

All this is to try to give you a realistic picture of where things are with Open Source Software (OSS) and SharpMap in particular. Now, I will give some input into your specific expectations, but it is, in light of the foregoing, just speculation until there is something actually working and up on the site.

1) GIS interfaces conforming to OGC specs.
This is what GeoAPI is. I know this is unusable at this time, but it's being worked on. I need it myself for my employer, so barring any unusual calamity, this should ship soon.

2) Replace Proj.Net with another projection library.
GeoAPI integration will allow this. Any wrapper to any existing projection library (e.g. Proj4) may not exist, and therefore would need to be done.

3) Clean factoring of GeoAPI interfaces.
SharpMap will only use the GeoAPI interfaces, and there will be no reference to NTS. As far as interoperability however, this seems like it would need to be tested, since even though it is designed they way you describe, the complexity is such that NTS may not integrate with an alternate GeoAPI implementation, since it depends on an internal implementation. So, for example, NTS will probably not be able to use a different implementation of GeoAPI.Geometries.IGeometry<TCoordinate> in order to compute a buffer. While this could conceivably be done, since the internal factoring of NTS is fairly clean, it is not planned.

4) Refactoring of IO classes.
I have done this. The version of NTS I am using has these removed. I have put WKT and WKB parsing into GeoAPI for now, although these could be refactored into interfaces with implementation in a separate project as well. No current plans for this, however, and no plans for handling GML or KML at the moment.

5) Data providers.
The separation of data providers is fairly complete, layer wise. The implementation of a number of providers (ShapeFile, in-memory) lives in the SharpMap project right now. This could change, but I know of no concrete plans to split it out. Other providers implement the interfaces, but live in other projects.

6) Encoders.
Not clear how this is different from data providers.

7) Rendering separation.
Rendering in v2.0 is already separate, and broken out into different assemblies.

8) No UI widgets.
I'm assuming you are talking about widgets other than the ones needed to render the geometries and paint the raster images. SharpMap v2.0 defines interfaces for the views, and the logic needed to present the data, but no UI widgets. This is how it should stay.

9) Geoprocessing independent of SharpMap / move data provider interfaces.
I've been thinking about doing this, since it makes sense. I can't do anything about it right now, however, since v2.0 Beta 2 very much needs to be done. I know this idea has been kicked around on IRC and the forums before, and I have been trying to conceive on how to design it with LINQ becoming ascendant. It appears easy at first glance, but if it's done like ADO.Net currently exists, we are basically ignoring the last 6 years of thinking on data access, and making it harder to use in the near to mid-term future.

10) Reliability
It's hard to answer this one, since there is no single request, but here are some points for everyone trying to follow along with the code...

- The code is on Google Code since CodePlex didn't support Subversion when we started the v2.0 branch. It still has suboptimal support (look at GeoAPI: I can't use SvnBridge with it anymore since SvnBridge doesn't handle character encoding correctly). SvnBridge is a Microsoft supported initiative, but they seem to be having a good deal of trouble getting it to work over the past 6 mo. of development. Not unlike SharpMap, it seems. ;)

- Refactoring: the main reason was to get all the above expectations met.

- There is no hard and fast time-line for completion. SharpMap contributors serve at the desire of their employers. If other work comes up for me or the others, we don't work on SharpMap. This is fairly typical. I made this mistake waiting on NDoc. I also made it on NAnt. Same for NCover. I wasn't paying these guys, so I was getting what I was paying for. I've learned to live with it, and know the risks, and only depend on what I have, not what is coming in any short-term.

- There are a number of developers. Of course Morten Nielsen, the original author, Christian, Diego, Ricardo, a few of my fellow employees, and other contributors (Bladewise, Joel, Dan, Bill Dollins). Morten doesn't contribute at this time, but that's how OSS happens. NUnit, for example, went through 3 (or 4) project heads, and along the way it progressed. After one developer stops, another starts. This is great in the long term, and nightmarish in the short.

- .Net dependency: MS finally released v2.0 SP1. SharpMap works with it, so .Net 3.5 isn't needed. I plan to make this more clear when I provide some update to the Beta 1 trunk next week. The reason for the dependency was for spatial queries on the FeatureDataTable to work. Only .Net 2.0 SP1 had the needed internal construct to get this to work.

Dec 20, 2007 at 11:13 AM
Edited Dec 20, 2007 at 11:18 AM
Clarification... we didn't loose the contract because we couldn't deliver a SharpMap solution in two weeks or two months or whatever. We lost it because our proposal was based upon an open source development effort over which it seemed we had little control or insight. They follow the SharpMap thread, and saw the several references to refactoring being completed next week... and it was never completed next week. Hey... I understand. They hear the same thing from me on my projects, and I'm NOT open source :) I could easily explain that we are waiting on 2.0 because of the integration of GeoAPI and NTS, and the refactoring of the ProjNet interfaces into GeoAPI, etc, etc. And, we now have a functioning build that is delivering great utility to them based upon those Nov 5 refactored components. It just can't display maps yet :) The problem was... I could not explain what was happening after Nov 5. I know that you are making changes to support generics. But I don't fully understand the benefit of that as we go down the road. Maybe I didn't get into hard stuff, but the Nov 5 API's have been working just fine for our needs. So, when I could not explain the development direction of the project, and they were already queasy over the .Net 3.5 decision (yeah... I know it is 2.1, but 3.5 had to be deployed to get 2.1) they got scared. And, we see so much potential for this effort, we were not willing to go with the commercial solution that they wanted us to change to.

That's fine... we'll have lot's of other opportunities :)

On the points... in general, I knew the answers to most of those "expectations", and it was an exercise to reinforce why I think this effort is going to be so good. That's the list I would go to a commercial vendor with. And I don't find any commercial product that can stand up to where the SharpMap/NTS/GeoAPI effort is headed. I appreciate the clarification on some of them. Comments on specific points:

3) Clean factoring of the GeoAPI interfaces -- Thank you for the heads up on NTS having dependencies on internal methods/structures of its geometry objects. We can work around that, and I may make it a personal goal to explore the performance implications of eliminating any dependencies on internal properties of the geoemtry objects. But it is a very low priority.

4) Removing the shapefile writers from NTS -- Thank you. Do you need us to start a new project that replaces that functionality? Or, the data provider implementations of SharpMap will serve everyone's needs?

8) No UI Widgets -- Your assumption is correct. Map Rendering certainly needs to be a part of SharpMap. A legend box, scale bar, etc, etc... while these would be nice, they don't need to be part of the main assembly and I should not nave to include them in my applications. And I believe that's where we either are or are headed, right?

9) You have mentined LINQ several times. Am I going to have to learn what LINQ is to be able to discusss the future of SharpMap? :)


I love what I see at the 30,000 foot view of this set of projects. We want to help. I see several references to "need more developers involved". I have asked several times about the areas where we can help, and I have received no direction. I understand that you can't have extra people jumping in to the refactoring area. So there is no place else where we can help in the mean time?

Right now, we've written a lot of code that uses the Nov 5 GeoAPI and NTS builds. I understand that this code will all have to change... and I have NO problem with that. I'm just chomping at the bit to be able to get these two projects in a state where we can make our code changes and start updating our test cases for exercising them in GeoProcessing functions. I don't care if stuff breaks. If we can just get an NTS that compiles, even if the GeoAPI and NTS interfaces will still be changing... I would be happy to start writing code that exercises the objects, and maintain the code through any future API changes.

And, yeah... I really want to see one of my 100,000 site maps rendered :)

Thank you for your efforts, sir.