Tips & Tricks for Spatial Data Harmonization

1 tips & Tricks for Spatial Data Harmonization Dr. Christine Giger Dr. Jan Schulze Althoff Overview Why is Spatial Data Harmonization still important or necessary? Tools & Methods tips & Tricks Conclusions INSPIRE Conference 2013, Florence - Dr. Christine Giger 2. Overview Why is Spatial Data Harmonization still important or necessary? Tools & Methods tips & Tricks Conclusions INSPIRE Conference 2013, Florence - Dr. Christine Giger 3. Provision of INSPIRE-compliant data All of the commercial SW vendors and many open source products offer off- the-shelf solutions to work with INSPIRE-compliant services All data providers deliver their data INSPIRE compliant Everything should be interoperable when using INSPIRE-compliant data INSPIRE Conference 2013, Florence - Dr. Christine Giger 4. BUT: Observed Problems (Software). Still various interoperability issues between different software systems data exported by system abc as INSPIRE compliant data cannot directly be used with system xyz some examples: The data is not valid against the schema The data and/or the schema cannot be imported into the favoured Spatial ETL-Tool or GIS ABC.

2 The data cannot be visualized in the ETL-Tool or GIS ABC. Data on different themes created by the same tool cannot be integrated or migrated The data cannot be migrated with other (non-INSPIRE). GML/XML data Etc. INSPIRE Conference 2013, Florence - Dr. Christine Giger 5. Causes for the problems Many GIS and : .. operate data-centred and are not schema- aware . Data is not validated in the production process (errors like missing attributes, non-declared elements). Implicit restrictions ( specific geometry representation required - <pos> vs. <poslist> or specific position of srs . Attribute).. use hard-coded namespaces and schema locations .. use a vendor-specific core schema .. use deprecated types . INSPIRE Conference 2013, Florence - Dr. Christine Giger 6. Observed Problems (Data provision). Often data is still delivered in different formats/structures Several XML-based formats, proprietary XML formats GML GML (in different flavors).

3 Schemas are huge (up to million lines). and (partly) complex ( 580 complex types and over 80 referenced schemas). INSPIRE Conference 2013, Florence - Dr. Christine Giger 7. Example: Overview on the structure of German topographic data (here: Streets). AbstractFeatureType AA_ObjektType hatDirektUnten AA_REOType AG_ObjekteMit istTeilVon GemeinsamerGeometrie AA_ZUSOType TA_CurveComponentType AX_Strasse AX_StrasseType AX_Strassenachse AX_StrassenachseType AX_Fahrbahnachse AX_FahrbahnachseType INSPIRE Conference 2013, Florence - Dr. Christine Giger 8. Overview Why is Spatial Data Harmonization still important or necessary? Tools & Methods tips & Tricks Conclusions INSPIRE Conference 2013, Florence - Dr. Christine Giger 9. Requirements for data Harmonization Interoperability between software minor adaptions (adding/removing attributes, changing namespaces, validating ). -> Scripting tasks.

4 Data transformation of delivered/. provided data major structural changes (extraction of elements, reclassification, grouping, ). -> Complex transformations . INSPIRE Conference 2013, Florence - Dr. Christine Giger 10. Requirements for tools to support data transformation Technical requirements (grouped/simplified). Read & Write XML Data Support namespaces & schema validation Support filtering on values, types, structures Support conditional statements Support group functions Support simple Spatial operations Support GML types directly INSPIRE Conference 2013, Florence - Dr. Christine Giger 11. Technology used in different projects Spatial ETL. Safe FME. Talend Spatial Data Integrator GeoKettle Humboldt Alignment Editor . Combinations of open toolsets XSLT Python GDAL. XQuery (incl. ExPath Geo Module). Observation: No or very few Spatial transformations are needed INSPIRE Conference 2013, Florence - Dr.

5 Christine Giger 12. XQuery - Overview Functional language to query and create XML. Official W3C standard aligned with XPath for adressing XML. XSLT as template language Increasing relevance and maturity esp. XML Databases (eXist DB, Marklogic, Oracle, MS, ..). Several Tools Execution Environments (Saxon, Zorba, Altova, ). Development Support(Eclipse XQDT, XML Spy, Oxygen, Stylus Studio, ). INSPIRE Conference 2013, Florence - Dr. Christine Giger 13. XQuery - Technically Functional language Execution as chain of functions Variety of predefined functions Standard functions for Strings, Numerics, Paths, . Extended functions for fulltext, geo operations, . External functions in C, Java, . XML oriented XPath based selection and filtering Native XML types & Schema aware Loops, conditional statements and grouping on XML. Collections Static & dynamic creation of XML elements INSPIRE Conference 2013, Florence - Dr.

6 Christine Giger 14. XQuery for Spatial Data Harmonization Pros Open standard; several implementations and tools Optimized for XML Processing (Schema aware;. collections/sequences and XML types). Modularization and external libraries ( ExPath Geo). Cons Programming language with specific syntax (steep learning curve). No direct geospatial support INSPIRE Conference 2013, Florence - Dr. Christine Giger 15. Overview Why is Spatial Data Harmonization still important or necessary? Tools & Methods tips & Tricks Conclusions INSPIRE Conference 2013, Florence - Dr. Christine Giger 16. Main Tip & Trick : use XQuery Trick: for data/schema reduction Simplifying data analysis and understanding of structures Speed up processing Tip: for data/schema transformation Encapsulate repetitive tasks in functions Build modules for common structures INSPIRE Conference 2013, Florence - Dr. Christine Giger 17.

7 Example 1: Data reduction by filtering of Featuretypes declare variable $input as element() := validate strict 1. {doc('file:///C:/ ')/gid:AX_Bestandsdatenauszug};. let $featureSet =. $input/enthaelt/wfs:FeatureCollection/gm l:featureMember 2. [name(child::*)='AX_Strasse' or name(child::*)='AX_Strassenachse' or name(child::*)='AX_Fahrbahnachse' ]. for $feature in $featureSet 3. return $feature 1. Define and validate an external file as datasource 2. Selection of XML elements by using XPath expression 3. Iterate the result and return data INSPIRE Conference 2013, Florence - Dr. Christine Giger 18. Example 2: Reclassification of values 1. switch ($strasse/aaa:widmung). case "1301" return attribute{ abc:roadType }{ highway }. 2. case "1303" return attribute{ abc:roadType }{ road }. default return element attribute{ abc:roadType }{ unknown }. 3. 1. Select the criteria 2. Decode the values and create according attributes 3.

8 Return default attribute value INSPIRE Conference 2013, Florence - Dr. Christine Giger 19. tips : Get started with XQuery Info: XQuery Spec: XQuery Tutorial: Environment: Zorba (XQuery Processor). Eclipse XQDT. INSPIRE Conference 2013, Florence - Dr. Christine Giger 20. XQuery Tools - XQDT. INSPIRE Conference 2013, Florence - Dr. Christine Giger 21. XQuery Tools - XQDT. Tip: Use the older Eclipse Indigo . (Eclipse Juno and Kepler fail on big XML Schemas). INSPIRE Conference 2013, Florence - Dr. Christine Giger 22. Overview Why is Spatial Data Harmonization still important or necessary? Tools & Methods tips & Tricks Conclusions INSPIRE Conference 2013, Florence - Dr. Christine Giger 23. Conclusions XQuery is an excellent, easy-to-use method to specify and execute transformations for XML/GML data Further possibilities for simplification, dependent on input and output schemas INSPIRE Conference 2013, Florence - Dr.

9 Christine Giger 24. Next Steps - XQuery Modules Simplifying Modules: Schema Modules: (Creation of (Creation of Feature, Point, Curve, Surface) CoreElements). (Harmonising Geometry, Simple Transf.). GML Base Modules: (XLink Schema). Helper: Tools (Type extension, UUID, ). External Calls INSPIRE Conference 2013, Florence - Dr. Christine Giger 25. Conclusions We are very much interested in exchanging experiences on the usage of XQuery for the transformation of Spatial data! Thank you for your attention! INSPIRE Conference 2013, Florence - Dr. Christine Giger 26.