In the past, I have posted about job trends for traditional programming languages as well as Web 2.0 programming languages. This is another part of the job trends series, the NoSQL job trends, which I started about 6 months ago. NoSQL data stores continue to be a hot topic as more services implement them for massive data. As was done for the previous job trends posts, I looked at the trends from Indeed and SimplyHired to see what technologies were garnering demand. I have continued to avoid including traditional RDBMS systems (MySQL, Oracle, DB2, and SQL Server) as they would skew the data.

You will notice that Hadoop is missing. Hadoop was not included, though HBase is part of it, because it skewed the results in a way that made the other trends hard to decipher. Hadoop is the clear leader in demand right now but more demand is appearing for specific components which makes the trends significantly easier to read. The tools that are included in this analysis are Cassandra, Redis, Voldemort, SimpleDB, CouchDB, MongoDB, HBase and Hypertable. One note on Cassandra is that there is a significant amount of noise in the trend data. This is due to Cassandra being a woman's name and appearing in a significant number of jobs. However, as opposed to my previous post on NoSQL jobs, the trend for Cassandra has changed and can be used in this analysis.

Let's start by looking the the basic trends from Indeed.

You can quickly see in this graph that there was a huge surge at the beginning of this year. This surge has continued throughout the year as NoSQL popularity has continued to grow. Cassandra has seen a huge amount of increasing demand this year. Now that Facebook and Digg use it in production systems, that growth is likely to continue unless significant problems are found. HBase was the leader previously, due to being a component of Hadoop, but there is a clear change in the trend. HBase demand basically hit a plateau due to the growth of Cassandra and MongoDB. MongoDB demand has also skyrocketed in the past 6 months, overtaking HBase in that span. CouchDB and Redis have also shown solid growth. SimpleDB has been on a roller coaster ride for the past year, but is currently trending well. Voldemort does not have a good long term trend, but the past few months have shown slight growth. Hypertable does not seem to be able to maintain demand as its growth has flatlined.

Now, what do the trends look like on SimplyHired.

Due to the shorter timeline, SimplyHired gives us a different view on these trends. Here we can see the general growth of the NoSQL niche from the beginning of the year more clearly. Similar to the Indeed trends, we can see Cassandra quickly growing and becoming the leader (besides Hadoop of course). The one thing that is not obvious in the Indeed trends is the large dip from May to June. This could just be due to seasonal hiring practices, but we do not have enough history to compare with. Otherwise, the growth trends for Cassandra, HBase, MongoDB, and CouchDB are similar to the Indeed trends. Redis and SimpleDB are showing much slower growth than the leaders. SimplyHired shows a much clearer lack of demand for Voldemort and Hypertable.

Finally, let's look at the relative growth trends from Indeed.

I say the same thing with every job trends post, I love the relative growth chart from Indeed. The past 6 months have seen huge growth in MongoDB. Granted MongoDB is one of the younger technologies in a young niche, but there is a lot of hype around it. That hype seems to be driving demand. Cassandra and HBase still show a ton of growth regardless of how much demand is already there. Interestingly, Redis shows hockey-stick growth, but that has not translated to the same demand as some of the other NoSQL offerings. CouchDB seems to be growing nicely, while SimpleDB needs some consistency. Voldemort and Hypertable trail in relative growth as well, which really symbolizes how little traction they have at this time.

Other NoSQL Resources

The initial list of tools was source from Rick Cattell's Scalable DataStores page, which also links to some excellent PDFs.

If you are interested in these types of jobs, take a look at this list of links about NoSQL required reading.

Related posts:

• NoSQL Job Trends March 2010
• Traditional Programming Language Job Trends
• Web & Scripting Programming Language Job Trends August 2010

The title of this post is a translation of an Arabic saying that my father quoted me throughout my childhood.

I have been teaching my NHibernate course these past few days, and I had come to realize that my approach for designing RDBMS based applications has gone a drastic change recently. I think that the difference in my view was brought home when I started getting angry about this model:

I mean, it is pretty much a classic, isn't it? But what really annoyed me was that all I had to do was look at this and know just how badly this is going to end up as when someone is going to try to show an order with its details. We are going to have, at least initially, 3 + N(order lines) queries. And even though this is a classic model, loading it efficiently is actually not that trivial. I actually used this model to show several different ways of eager loading. And remember, this model is actually a highly simplified representation of what you'll need in real projects.

I then came up with a model that I felt was much more palatable to me:

And looking at it, I had an interesting thought. My problem with the model started because I got annoyed by how many tables were involved in dealing with Show Order, but the end result also reminded me of something, Root Aggregates in DDDs. Now, since my newfound sensitivity about this has been based on my experiences with RavenDB, I found it amusing that I explicitly modeled documents in RavenDB after Root Aggregates in DDD, then went the other way (reducing queries > Root Aggregates) with modeling in RDBMS).

The interesting part is that once you start thinking like this, you end up with a lot of additional reasons why you actually want that. (If the product price changed, it doesn't affect the order, for example).

If you think about it, normalization in RDBMS had such a major role because storage was expensive. It made sense to try to optimize this with normalization. In essence, normalization is compressing the data, by taking the repeated patterns and substituting them with a marker. There is also another issue, when normalization came out, the applications being being were far different than the type of applications we build today. In terms of number of users, time that you had to process a single request, concurrent requests, amount of data that you had to deal with, etc.

Under those circumstances, it actually made sense to trade off read speed for storage. In today's world? I don't think that it hold as much.

The other major benefit of normalization, which took extra emphasis when the reduction in storage became less important as HD sizes grew, is that when you state a fact only once, you can modify it only once.

Except there is a large set of scenarios where you don't want to do that. Take invoices as a good example. In the case of the order model above, if you changed the product name from Thingamajig to Foomiester, that is going to be mighty confusing for me when I look at that order and have no idea what that thing was. What about the name of the customer? Think about the scenarios in which someone changes their name (marriage is most common one, probably). If a woman orders a book under her maiden name, then changes her name after she married, what is supposed to show on the order when it is displayed? If it is the new name, that person didn't exist at the time of the order.

Obviously, there are counter examples, which I am sure the comments will be quick to point out.

But it does bear thinking about, and my default instinct to apply 3rd normal form has been muted once I realized this. I now have a whole set of additional questions that i ask about every piece of information that I deal with.

Earlier Listing is Expanded by More than 30%

At the beginning of this year Structured Dynamics assembled a listing of ontology building tools at the request of a client. That listing was presented as The Sweet Compendium of Ontology Building Tools. Now, again because of some client and internal work, we have researched the space again and updated the listing [1].

All new tools are marked with <New> (new only means newly discovered; some had yet to be discovered in the prior listing). There are now a total of 185 tools in the listing, 31 of which are recently new, and 45 added at various times since the first release.

Comprehensive Ontology Tools

• Altova SemanticWorks is a visual RDF and OWL editor that auto-generates RDF/XML or nTriples based on visual ontology design. No open source version available
• Amine is a rather comprehensive, open source platform for the development of intelligent and multi-agent systems written in Java. As one of its components, it has an ontology GUI with text- and tree-based editing modes, with some graph visualization
• The Apelon DTS (Distributed Terminology System) is an integrated set of open source components that provides comprehensive terminology services in distributed application environments. DTS supports national and international data standards, which are a necessary foundation for comparable and interoperable health information, as well as local vocabularies. Typical applications for DTS include clinical data entry, administrative review, problem-list and code-set management, guideline creation, decision support and information retrieval.. Though not strictly an ontology management system, Apelon DTS has plug-ins that provide visualization of concept graphs and related functionality that make it close to a complete solution
• DOME is a programmable XML editor which is being used in a knowledge extraction role to transform Web pages into RDF, and available as Eclipse plug-ins. DOME stands for DERI Ontology Management Environment
• FlexViz is a Flex-based, Protg-like client-side ontology creation, management and viewing tool; very impressive. The code is distributed from Sourceforge; there is a nice online demo available; there is a nice explanatory paper on the system, and the developer, Chris Callendar, has a useful blog with Flex development tips
• Knoodl facilitates community-oriented development of OWL based ontologies and RDF knowledge bases. It also serves as a semantic technology platform, offering a Java service-based interface or a SPARQL-based interface so that communities can build their own semantic applications using their ontologies and knowledgebases. It is hosted in the Amazon EC2 cloud and is available for free; private versions may also be obtained. See especially the screencast for a quick introduction
• The NeOn toolkit is a state-of-the-art, open source multi-platform ontology engineering environment, which provides comprehensive support for the ontology engineering life-cycle. The v2.3.0 toolkit is based on the Eclipse platform, a leading development environment, and provides an extensive set of plug-ins covering a variety of ontology engineering activities. You can add these plug-ins or get a current listing from the built-in updating mechanism
• ontopia is a relative complete suite of tools for building, maintaining, and deploying Topic Maps-based applications; open source, and written in Java. Could not find online demos, but there are screenshots and there is visualization of topic relationships
• Protg is a free, open source visual ontology editor and knowledge-base framework. The Protg platform supports two main ways of modeling ontologies via the Protg-Frames and Protg-OWL editors. Protg ontologies can be exported into a variety of formats including RDF(S), OWL, and XML Schema. There are a large number of third-party plugins that extends the platform's functionality
  • Protg Plugin Library frequently consult this page to review new additions to the Protg editor; presently there are dozens of specific plugins, most related to the semantic Web and most open source
  • Collaborative Protg is a plug-in extension of the existing Protg system that supports collaborative ontology editing as well as annotation of both ontology components and ontology changes. In addition to the common ontology editing operations, it enables annotation of both ontology components and ontology changes. It supports the searching and filtering of user annotations, also known as notes, based on different criteria. There is also an online demo
  • <New>Web Protg is an online version of Protg attempting to capture all of the native functionality; still under development
• <New>Sigma is open source knowledge engineering environment that includes ontology mapping, theorem proving, language generation in multiple languages, browsing, OWL read/write, and analysis. It includes the Suggested Upper Merged Ontology (SUMO), a comprehensive formal ontology. It's under active development and use
• TopBraid Composer is an enterprise-class modeling environment for developing Semantic Web ontologies and building semantic applications. Fully compliant with W3C standards, Composer offers comprehensive support for developing, managing and testing configurations of knowledge models and their instance knowledge bases. It is based on the Eclipse IDE. There is a free version (after registration) for small ontologies
• <New>TwoUse Toolkit is an implementation of current OMG and W3C standards for developing ontology-based software models and model-based OWL2 ontologies, largely based around UML. There are a variety of tools, including graphics editors, with more to come
• <New>Wandora is a topic maps engine written in Java with support for both in-memory topic maps and persisting topic maps in MySQL and SQL Server. It also contains an editor and a publishing system, and has support for automatic classification. It can read OBO, RDF(S), and many other formats, and can export topic maps to various graph formats. There is also a web-based topic maps browser, and graphical visualization.

Not Apparently in Active Use

• Adaptiva is a user-centred ontology building environment, based on using multiple strategies to construct an ontology, minimising user input by using adaptive information extraction
• Exteca is an ontology-based technology written in Java for high-quality knowledge management and document categorisation, including entity extraction. Though code is still available, no updates have been provided since 2006. It can be used in conjunction with search engines
• IODT is IBM's toolkit for ontology-driven development. The toolkit includes EMF Ontolgy Definition Metamodel (EODM), EODM workbench, and an OWL Ontology Repository (named Minerva)
• KAON is an open-source ontology management infrastructure targeted for business applications. It includes a comprehensive tool suite allowing easy ontology creation and management and provides a framework for building ontology-based applications. An important focus of KAON is scalable and efficient reasoning with ontologies
• Ontolingua provides a distributed collaborative environment to browse, create, edit, modify, and use ontologies. The server supports over 150 active users, some of whom have provided us with descriptions of their projects. Provided as an online service; software availability not known.

Vocabulary Prompting Tools

• AlchemyAPI from Orchestr8 provides an API based application that uses statistical and natural language processing methods. Applicable to webpages, text files and any input text in several languages
• BooWa is a set expander for any language (formerly known as SEALS); developed by RC Wang of Carnegie Mellon
• Google Keywords allows you to enter a few descriptive words or phrases or a site URL to generate keyword ideas
• Google Sets for automatically creating sets of items from a few examples
• Open Calais is free limited API web service to automatically attach semantic metadata to content, based on either entities (people, places, organizations, etc.), facts (person x' works for company y'), or events (person z' was appointed chairman of company y' on date x'). The metadata results are stored centrally and returned to you as industry-standard RDF constructs accompanied by a Globally Unique Identifier (GUID)
• Query-by-document from BlogScope has a nice phrase extraction service, with a choice of ranking methods. Can also be used in a Firefox plug-in (not texted with 3.5+)
• SemanticHacker (from Textwise) is an API that does a number of different things, including categorization, search, etc. By using concept tags', the API can be leveraged to generate metadata or tags for content
• TagFinder is a Web service that automatically extracts tags from a piece of text. The tags are chosen based on both statistical and linguistic analysis of the original text
• Tagthe.net has a demo and an API for automatic tagging of web documents and texts. Tags can be single words only. The tool also recognizes named entities such as people names and locations
• TermExtractor extracts terminology consensually referred in a specific application domain. The software takes as input a corpus of domain documents, parses the documents, and extracts a list of syntactically plausible terms (e.g. compounds, adjective-nouns, etc.)
• TermFinder uses Poisson statistics, the Maximum Likelihood Estimation and Inverse Document Frequency between the frequency of words in a given document and a generic corpus of 100 million words per language; available for English, French and Italian
• TerMine is an online and batch term extractor that emphasizes part of speech (POS) and n-gram (phrase extraction). TerMine is the terminological management system with the C-Value term extraction and AcroMine acronym recognition integrated
• Topia term extractor is a part-of-speech and frequency based term extraction tool implemented in python. Here is a term extraction demo based on this tool
• Topicalizer is a service which automatically analyses a document specified by a URL or a plain text regarding its word, phrase and text structure. It provides a variety of useful information on a given text including the following: Word, sentence and paragraph count, collocations, syllable structure, lexical density, keywords, readability and a short abstract on what the given text is about

• TrMExtractor does glossary extraction on pure text files for either English or Hungarian

• Wikify! is a system to automatically wikify a text by adding Wikipedia-like tags throughout the document. The system extracts keywords and then disambiguates and matches them to their corresponding Wikipedia definition
• Yahoo! Placemaker is a freely available geoparsing Web service. It helps developers make their applications location-aware by identifying places in unstructured and atomic content feeds, web pages, news, status updates and returning geographic metadata for geographic indexing and markup
• Yahoo! Term Extraction Service is an API to Yahoo's term extraction service, as well as many other APIs and services in a variety of languages and for a variety of tasks; good general resource. The service has been reported to be shut down numerous times, but apparently is kept alive due to popular demand.

Initial Ontology Development

• COE COE (CmapTools Ontology Editor) is a specialized version of the CmapTools from IMHC. COE and its CmapTools parent is based on the idea of concept maps. A concept map is a graph diagram that shows the relationships among concepts. Concepts are connected with labeled arrows, with the relations manifesting in a downward-branching hierarchical structure. COE is an integrated suite of software tools for constructing, sharing and viewing OWL encoded ontologies based on these constructs
• Conzilla2 is a second generation concept browser and knowledge management tool with many purposes. It can be used as a visual designer and manager of RDF classes and ontologies, since its native storage is in RDF. It also has an online collaboration server [apparently last updated in 2008]
• http://diagramic.com/ has an online Flex network graph demo, which also has a neat facility for quick entry and visualization of relationships; mostly small scale; pretty cool. Does not appear to be code available anywhere
• <New>DL-Learner is a tool for learning OWL class expressions from examples and background knowledge. It extends Inductive Logic Programming (ILP) to Description Logics and the Semantic Web. DL-Learner now has a flexible component based design, which allows to extend it easily with new learning algorithms, learning problems, reasoners, and supported background knowledge sources. A new type of supported knowledge sources are SPARQL endpoints, where DL-Learner can extract knowledge fragments, which enables learning classes even on large knowledge sources like DBpedia, and includes an OWL API reasoner interface and Web service interface.
• DogmaModeler is a free and open source, ontology modeling tool based on ORM. The philosophy of DogmaModeler is to enable non-IT experts to model ontologies with a little or no involvement of an ontology engineer; project is quite old, but the software is still available and it may provide some insight into naive ontology development
• Erca is a framework that eases the use of Formal and Relational Concept Analysis, a neat clustering technique. Though not strictly an ontology tool, Erca could be implemented in a work flow that allows easy import of formal contexts from CSV files, then algorithms that computes the concept lattice of the formal contexts that can be exported as dot graphs (or in JPG, PNG, EPS and SVG formats). Erca is provided as an Eclipse plug-in
• GraphMind is a mindmap editor for Drupal. It has the basic mindmap features and some Drupal specific enhancements. There is a quick screencast about how GraphMind looks like and what is does. The Flex source is also available from Github
• <New>H-Maps is a commercial suite of tools for building topic maps applications, consisting of a topic maps engine and server, a mapping framework for converting from legacy data, and a navigator for visualizing data. It is typically used in bioinformatics (drug discovery and research, toxicological studies, etc), engineering (support and expert systems), and for integration of hetereogeneous data. It supports the XTM 1.0 and TMAPI 1.0 specifications
• irON using spreadsheets, via its notation and specification. Spreadsheets can be used for initial authoring, esp if the irON guidelines are followed. See further this case study of Sweet Tools in a spreadsheet using irON (commON)
• ITM T3 stands for Terminology, Thesaurus, Taxonomy, Metadata dictionary. ITM T3 includes a range of functions for managing enterprise shareable multilingual domain-specific taxonomies, thesaurus, terminologies in a unified way. It uses XML, SKOS and RDF standards. Commercial; from Mondeca
• <New>JXML2OWL API is a library for mapping XML schemas to OWL Ontologies on the JAVA platform. It creates an XSLT which transforms instances of the XML schema into instances of the OWL ontology. JXML2OWL Mapper is GUI application using the JXML2OWL API
• MindRaider is Semantic Web outliner. It aims to connect the tradition of outline editors with emerging technologies. MindRaider mission is to organize not only the content of your hard drive but also your cognitive base and social relationships in a way that enables quick navigation, concise representation and inferencing
• <New>Neologism is a simple web-based RDF Schema vocabulary editor and publishing system. Use it to create RDF classes and properties, which are needed to publish data on the Semantic Web. Its main goal is to dramatically reduce the time required to create, publish and modify vocabularies for the Semantic Web. It is written in PHP and built on the Drupal platform. Neologism is currently in alpha
• <New>OCS Ontology Creation System is software to develop ontologies in cooperative way with a graphical interface
• RDF123 is an application and web service for converting data in simple spreadsheets to an RDF graph. Users control how the spreadsheet's data is converted to RDF by constructing a graphical RDF123 template that specifies how each row in the spreadsheet is converted as well as metadata for the spreadsheet and its RDF translation
• <New>ROC (Rapid Ontology Construction) is a tool that allows domain experts to quickly build a basic vocabulary for their domain, re-using existing terminology whenever possible. How this works is that the ROC tool asks the domain expert for a set of keywords that are core' terms of the domain, and then queries remote sources for concepts matching those terms. These are then presented to the user, who can select terms from the list, find relations to other terms, and expand the set of terms and relations, iteratively. The resulting vocabulary (or proto-ontology', basically a SKOS-like thesaurus) can be used as is, or can be used as input for a knowledge engineer to base a more comprehensive domain ontology on. Interface triples-oriented, not graphical.
• Topincs is a Topic Map authoring software that allows groups to share their knowledge over the web. It makes use of a variety of modern technologies. The most important are Topic Maps, REST and Ajax. It consists of three components: the Wiki, the Editor, and the Server. The servier requires AMP; the Editor and Wiki are based on browser plug-ins.

Ontology Editing

• First, see all of the Comprehensive Tools and Ontology Development listings above
• Anzo for Excel includes an (RDFS and OWL-based) ontology editor that can be used directly within Excel. In addition to that, Anzo for Excel includes the capability to automatically generate an ontology from existing spreadsheet data, which is very useful for quick bootstrapping of an ontology
• <New>ATop is a topic map browser and editor written in Java and supports the XTM 1.0 specification; project has not been updated since 2008
• Hozo is an ontology visualization and development tool that brings version control constructs to group ontology development; limited to a prototype, with no online demo
• Lexaurus Editor is for off-line creation and editing of vocabularies, taxonomies and thesauri. It supports import and export in Zthes and SKOS XML formats, and allows hierarchical / poly-hierarchical structures to be loaded for editing, or even multiple vocabularies to be loaded simultaneously, so that terms from one taxonomy can be re-used in another, using drag and drop. Not available in open source
• Model Futures OWL Editor combines simple OWL tools, featuring UML (XMI), ErWin, thesaurus and imports. The editor is tree-based and has a navigator tool for traversing property and class-instance relationships. It can import XMI (the interchange format for UML) and Thesaurus Descriptor (BT-NT XML), and EXPRESS XML files. It can export to MS Word.
• <New>OBO-Edit is an open source ontology editor written in Java. OBO-Edit is optimized for the OBO biological ontology file format. It features an easy to use editing interface, a simple but fast reasoner, and powerful search capabilities
• <New>Onotoa is an Eclipse-based ontology editor for topic maps. It has a graphical UML-like interface, an export function for the current TMCL-draft and a XTM export
• OntoTrack is a browsing and editing ontology authoring tool for OWL Lite. It combines a sophisticated graphical layout with mouse enabled editing features optimized for efficient navigation and manipulation of large ontologies
• OWLViz is an attractive visual editor for OWL and is available as a Protg plug-in
• PoolParty is a triple store-based thesaurus management environment which uses SKOS and text extraction for tag recommendations. See further this manual, which describes more fully the system's functionality. Also, there is a PoolParty Web service that enables a Zthes thesaurus in XML format to be uploaded and converted to SKOS (via skos:Concepts)
• SKOSEd is a plugin for Protege 4 that allows you to create and edit thesauri (or similar artefacts) represented in the Simple Knowledge Organisation System (SKOS).
• TemaTres is a Web application to manage controlled vocabularies, taxonomies and thesaurus. The vocabularies may be exported in Zthes, Skos, TopicMap, etc.
• ThManager is a tool for creating and visualizing SKOS RDF vocabularies. ThManager facilitates the management of thesauri and other types of controlled vocabularies, such as taxonomies or classification schemes
• Vitro is a general-purpose web-based ontology and instance editor with customizable public browsing. Vitro is a Java web application that runs in a Tomcat servlet container. With Vitro, you can: 1) create or load ontologies in OWL format; 2) edit instances and relationships; 3) build a public web site to display your data; and 4) search your data with Lucene. Still in somewhat early phases, with no online demos and with minimal interfaces.
• <New>Vocab Editor is an RDF/OWL/SKOS vocabulary-diagram editor. It has both client- (Javascript) and server-side (Python) implmentations. It is open source with a demo. There is a blog (Spanish) and online sample vocabulary app editor.

Not Apparently in Active Use

• Omnigator The Omnigator is a form-based manipulaton tool centered on Topic Maps, though it enables the loading and navigation of any conforming topic map in XTM, HyTM, LTM or RDF formats. There is a free evaluation version.
• OntoGen is a semi-automatic and data-driven ontology editor focusing on editing of topic ontologies (a set of topics connected with different types of relations). The system combines text-mining techniques with an efficient user interface. It requires .Net.
• OntoLight is a set of software modules for: transforming raw ontology data for several ontologies from their specific formats into a unifying light-weight ontology format, grounding the ontology and storing it into grounded ontology format, populating grounded ontologies with new instance data, and creating mappings between grounded ontologies; includes Cyc. Download no longer available. See http://analytics.ijs.si/~blazf/papers/Context_SiKDD07.pdf and http://www.neon-project.org/web-content/index.php?option=com_weblinks&task=view&catid=17&id=52 or http://www.neon-project.org/web-content/index.php?option=com_weblinks&catid=21&Itemid=73
• OWL-S-editor is an editor for the development of services in OWL-S, with graphical, WSDL and import/export support
• ReTAX+ is an aide to help a taxonomist create a consistent taxonomy and in particular provides suggestions as to where a new entity could be placed in the taxonomy whilst retaining the integrity of the revised taxonomy (c.f., problems in ontology modelling)
• SWOOP is a lightweight ontology editor. (Swoop is no longer under active development at mindswap. Continuing development can be found on SWOOP's Google Code homepage at http://code.google.com/p/swoop/)
• WebOnto supports the browsing, creation and editing of ontologies through coarse grained and fine grained visualizations and direct manipulation.

Ontology Mapping

• <New>The Alignment API is an API and implementation for expressing and sharing ontology alignments. The correspondences between entities (e.g., classes, objects, properties) in ontologies is called an alignment. The API provides a format for expressing alignments in a uniform way. The goal of this format is to be able to share on the web the available alignments. The format is expressed in RDF, so it is freely extensible. The Alignment API itself is a Java description of tools for accessing the common format. It defines four main interfaces (Alignment, Cell, Relation and Evaluator).
• COMA++ is a schema and ontology matching tool with a comprehensive infrastructure. Its graphical interface supports a variety of interaction
• ConcepTool is a system to model, analyse, verify, validate, share, combine, and reuse domain knowledge bases and ontologies, reasoning about their implication
• <New>MapOnto is a research project aiming at discovering semantic mappings between different data models, e.g, database schemas, conceptual schemas, and ontologies. So far, it has developed tools for discovering semantic mappings between database schemas and ontologies as well as between different database schemas. The Protege plug-in is still available, but appears to be for older versions
• MatchIT automates and facilitates schema matching and semantic mapping between different Web vocabularies. MatchIT runs as a stand-alone or plug-in Eclipse application and can be integrated with popular third party applications. MatchIT's uses Adaptive Lexicon as an ontology-driven dictionary and thesaurus of English language terminology to quantify and ank the semantic similarity of concepts. It apparently is not available in open source
• myOntology is used to produce the theoretical foundations, and deployable technology for the Wiki-based, collaborative and community-driven development and maintenance of ontologies instance data and mappings
• OLA/OLA2 (OWL-Lite Alignment) matches ontologies written in OWL. It relies on a similarity combining all the knowledge used in entity descriptions. It also deal with one-to-many relationships and circularity in entity descriptions through a fixpoint algorithm
• Potluck is a Web-based user interface that lets casual usersthose without programming skills and data modeling expertisemash up data themselves. Potluck is novel in its use of drag and drop for merging fields, its integration and extension of the faceted browsing paradigm for focusing on subsets of data to align, and its application of simultaneous editing for cleaning up data syntactically. Potluck also lets the user construct rich visualizations of data in-place as the user aligns and cleans up the data.
• PRIOR+ is a generic and automatic ontology mapping tool, based on propagation theory, information retrieval technique and artificial intelligence model. The approach utilizes both linguistic and structural information of ontologies, and measures the profile similarity and structure similarity of different elements of ontologies in a vector space model (VSM).
• <New>S-Match takes any two tree like structures (such as database schemas, classifications, lightweight ontologies) and returns a set of correspondences between those tree nodes which semantically correspond to one another.
• Vine is a tool that allows users to perform fast mappings of terms across ontologies. It performs smart searches, can search using regular expressions, requires a minimum number of clicks to perform mappings, can be plugged into arbitrary mapping framework, is non-intrusive with mappings stored in an external file, has export to text files, and adds metadata to any mapping. See also http://sourceforge.net/projects/vine/.

Not Apparently in Active Use

• ASMOV (Automated Semantic Mapping of Ontologies with Validation) is an automatic ontology matching tool which has been designed in order to facilitate the integration of heterogeneous systems, using their data source ontologies
• Chimaera is a software system that supports users in creating and maintaining distributed ontologies on the web. Two major functions it supports are merging multiple ontologies together and diagnosing individual or multiple ontologies
• CMS (CROSI Mapping System) is a structure matching system that capitalizes on the rich semantics of the OWL constructs found in source ontologies and on its modular architecture that allows the system to consult external linguistic resources
• ConRef is a service discovery system which uses ontology mapping techniques to support different user vocabularies
• DRAGO reasons across multiple distributed ontologies interrelated by pairwise semantic mappings, with a vision of peer-to-peer mapping of many distributed ontologies on the Web. It is implemented as an extension to an open source Pellet OWL Reasoner
• Falcon-AO (Finding, aligning and learning ontologies) is an automatic ontology matching tool that includes the three elementary matchers of String, V-Doc and GMO. In addition, it integrates a partitioner PBM to cope with large-scale ontologies
• FOAM is the Framework for ontology alignment and mapping. It is based on heuristics (similarity) of the individual entities (concepts, relations, and instances)
• hMAFRA (Harmonize Mapping Framework) is a set of tools supporting semantic mapping definition and data reconciliation between ontologies. The targeted formats are XSD, RDFS and KAON
• IF-Map is an Information Flow based ontology mapping method. It is based on the theoretical grounds of logic of distributed systems and provides an automated streamlined process for generating mappings between ontologies of the same domain
• LILY is a system matching heterogeneous ontologies. LILY extracts a semantic subgraph for each entity, then it uses both linguistic and structural information in semantic subgraphs to generate initial alignments. The system is presently in a demo version only
• MAFRA Toolkit the Ontology MApping FRAmework Toolkit allows users to create semantic relations between two (source and target) ontologies, and apply such relations in translating source ontology instances into target ontology instances
• OntoEngine is a step toward allowing agents to communicate even though they use different formal languages (i.e., different ontologies). It translates data from a source ontology to a target
• OWLS-MX is a hybrid semantic Web service matchmaker. OWLS-MX 1.0 utilizes both description logic reasoning, and token based IR similarity measures. It applies different filters to retrieve OWL-S services that are most relevant to a given query
• RiMOM (Risk Minimization based Ontology Mapping) integrates different alignment strategies: edit-distance based strategy, vector-similarity based strategy, path-similarity based strategy, background-knowledge based strategy, and three similarity-propagation based strategies
• semMF is a flexible framework for calculating semantic similarity between objects that are represented as arbitrary RDF graphs. The framework allows taxonomic and non-taxonomic concept matching techniques to be applied to selected object properties
• Snoggle is a graphical, SWRL-based ontology mapper. Snoggle attempts to solve the ontology mapping problem by providing a graphical user interface (similar to which of the Microsoft Visio) to guide the process of ontology vocabulary alignment. In Snoggle, user-defined mappings can be serialized into rules, which is expressed using SWRL
• Terminator is a tool for creating term to ontology resource mappings (documentation in Finnish).

Ontology Visualization/Analysis

Though all are not relevant, see my post from a couple of years back on large-scale RDF graph software.

• Social network graphing tools (many covered elsewhere)
• Cytoscape is a bioinformatics software platform for visualizing molecular interaction networks and integrating these interactions with gene expression profiles and other state data; I have also written specifically about Cytoscape's use in UMBEL
  • RDFScape is a project that brings Semantic Web features to the popular Systems Biology software Cytoscape
  • NetworkAnalyzer performs analysis of biological networks and calculates network topology parameters including the diameter of a network, the average number of neighbors, and the number of connected pairs of nodes. It also computes the distributions of more complex network parameters such as node degrees, average clustering coefficients, topological coefficients, and shortest path lengths. It displays the results in diagrams, which can be saved as images or text files; used by SD
• Graphl is a tool for collaborative editing and visualisation of graphs, representing relationships between resources or concepts of the real world. Graphl may be thought of as a visual wiki, a place where everybody can contribute to a shared repository of knowledge
• <New>Graphviz is open source graph visualization software. It has several main graph layout programs. It also has web and interactive graphical interfaces, and auxiliary tools, libraries, and language bindings.
• <New>GrOWL is an ontology visualizer and editor. The layout of the GrOWL graph can be defined automatically or loaded from a separate style sheet. GrOWL implements configurable filters that can transform the display by simplifying it, hiding concepts and relationships that have no descriptions associated, or perform more complex translations. Concepts can be stored in ontologies with extensive annotations to provide documentation. GrOWL shows these annotation as tooltips, and supports complex HTML and links within them. The GrOWL browser can be used inside a web browser or as a stand-alone application. When used inside a browser, it supports Javascript interaction so that it can be used as a concept chooser with implementation-defined operations.
• igraph is a free software package for creating and manipulating undirected and directed graphs
• Network Workbench is a very complex, comprehensive; Swiss Army Knife
• NetworkX Python; very clean
• <New>OntoGraf, a Protege 4 plug-in, gives support for interactively navigating the relationships of your OWL ontologies. Various layouts are supported for automatically organizing the structure of your ontology. Different relationships are supported: subclass, individual, domain/range object properties, and equivalence. Relationships and node types can be filtered.
• <New>OWL2Prefuse is a Java package which creats Prefuse graphs and trees from OWL files (and Jena OntModels). It takes care of converting the OWL data structure to the Prefuse datastructure. This makes it is easy for developers, to use the Prefuse graphs and trees into their Semantic Web applications.
• <New>RDF Gravity is a tool for visualising RDF/OWL Graphs/ ontologies. RDF Gravity is implemented by using the JUNG Graph API and Jena semantic web toolkit. Its main features are:
  • Graph Visualization
  • Global and Local Filters (enabling specific views on a graph)
  • Full text Search
  • Generating views from RDQL Queries
  • Visualising multiple RDF files
• Stanford Network Analysis Package (SNAP) is a general purpose network analysis and graph mining library. It is written in C++ and easily scales to massive networks with hundreds of millions of nodes
• Social Networks Visualizer (SocNetV) is a flexible and user-friendly tool for the analysis and visualization of Social Networks. It lets you construct networks (mathematical graphs) with a few clicks on a virtual canvas or load networks of various formats (GraphViz, GraphML, Adjacency, Pajek, UCINET, etc) and modify them to suit your needs. SocNetV also offers a built-in web crawler, allowing you to automatically create networks from all links found in a given initial URL
• Tulip may be incredibly strong
  • quite active (but not much online stuff): http://sourceforge.net/projects/auber/files/
• Springgraph component for Flex
• VizierFX is a Flex library for drawing network graphs. The graphs are laid out using GraphViz on the server side, then passed to VizierFX to perform the rendering. The library also provides the ability to run ActionScript code in response to events on the graph, such as mousing over a node or clicking on it.
• <New>VUE (Visual Understanding Environment) is an open source project focused on creating flexible tools for managing and integrating digital resources in support of teaching, learning and research. VUE provides a flexible visual environment for structuring, presenting, and sharing digital information.
• <New>yEd is a diagram editor that can be used to quickly and effectively generate high-quality drawings of diagrams. It can support OWL imports.
• <New>ZGRViewer is a graph visualizer implemented in Java and based upon the Zoomable Visual Transformation Machine. It is specifically aimed at displaying graphs expressed using the DOT language from AT&T GraphViz and processed by programs dot, neato or others such as twopi. ZGRViewer is designed to handle large graphs, and offers a zoomable user interface (ZUI), which enables smooth zooming and easy navigation in the visualized structure.

Miscellaneous Ontology Tools

• Apolda (Automated Processing of Ontologies with Lexical Denotations for Annotation) is a plugin (processing resource) for GATE (http://gate.ac.uk/). The Apolda processing resource (PR) annotates a document like a gazetteer, but takes the terms from an (OWL) ontology rather than from a list
• <New>Gloze is a XML to RDF, RDF to XML, and XSD to OWL mapping tool based on Jena; see also http://jena.hpl.hp.com/juc2006/proceedings/battle/paper.pdf . See also http://jena.sourceforge.net/contrib/contributions.html
• <New>Hoolet is an implementation of an OWL-DL reasoner that uses a first order prover. The ontology is translated to collection of axioms (in an obvious way based on the OWL semantics) and this collection of axioms is then given to a first order prover for consistency checking.
• LexiLink is a tool for building, curating and managing multiple lexicons and ontologies in one enterprise-wide Web-based application. The core of the technology is based on RDF and OWL
• ontology owl ontologies based web ontology ontologies based tool semantic data editor system source knowledge mapping tools protg provides text development open java used service topic interface user software graph language supports different plug maps written online graphs domain applications information application browser terms extraction including mappings library relationships framework automatically support skos management create enables allows files uses visual visualization project format graphical comprehensive platform vocabularies vocabulary analysis multiple formats model toolkit concepts includes similarity environment network concept listing simple thesaurus building server given search large active python classes relations sources alignment matching distributed structure demo variety content term schema networks export part tags learning properties metadata creating document easy apparently classification systems code creation free version terminology main reasoner features modeling instances schemas engineering automatic eclipse flex implementation linguistic extracts collaborative several jena resources managing components users prolog flexible designed client module implemented growl complex component generating import integrated html requires bases taxonomies taxonomy swoop tree protege spreadsheet external transform entity entities constructs quick generate google allowing plugin developed browsing languages english instance class offers formal documents obda design alignments models runs annotation manipulation expressed repository driven perform strategy defined graphviz input target extension first node addition provided database representation provide hierarchical piece supported current list versions initial sourceforge build takes developers resource standards suite social processing navigation technology modify descriptions editors keywords converting drupal specification words services wiki open source semantic web topic maps web service web based ontology mapping owl ontology owl ontologies ontology editor based ontology web ontology ontology based knowledge bases source ontology plug ins term extraction ontology development rdf xml ontology building user interface mapping framework owl api text files ontology engineering database schemas based strategy ontology management ontology browser ontology language multiple ontologies distributed ontologies topic map semantic mappings semantic mapping visual ontology extraction service rdf schema background knowledge tool based knowledge sources automatic ontology graph visualization sourceforge net language owl owl lite ontology creation online demo ontology matching data sources xml schema building tools music ontology owl files web toolkit matching tool neon project web content content index instance data index php discovering semantic online tool grounded ontologies ontology manager objects properties semantic similarity stand alone vocabulary importer alignment api external vocabulary different data entity descriptions net projects rdf gravity main features web applications prefuse graphs used inside web browser network analysis social networks first order order prover processing resource user friendly obda ontology object properties similarity based data source source ontologies managing multiple allows users structural information swi prolog owl reasoner ontology instances ontology format based tool large scale given ontology mapping tool excel includes formats including third party ontology components modeling ontologies online demos life cycle based applications ontology changes under active publishing system knowledge management maps engine w3c standards active development web ontologies comprehensive support engineering environment provides comprehensive information retrieval apelon dts tree based ontology tools owl editor management system web pages based interface semantic applications based ontologies owl based eclipse plug management environment ontology driven driven development simple knowledge web application protg plug ontology file ontology visualization vocabularies taxonomies controlled vocabularies general purpose topic ontologies ontology data server side diagram editor web site rdf owl editor written rdf graph linguistic analysis application domain phrase extraction carnegie mellon ontology repository based application term extractor given text jxml2owl api graphical interface formal contexts component based management tool rdf classes several ontologies web ontology language ontology building tools open source ontology ontology matching tool similarity based strategy based ontology mapping first order prover web based tool external vocabulary importer under active development automatic ontology matching mapping tool based web based ontology based ontology editor web content index topic maps engine discovering semantic mappings term extraction service ]]>AI3:::Adaptive Information
  First shared by - robdiana
  syndication+ 0 | Search 1 | Shares 1
  
  

  

  Earlier Listing is Expanded by More than 30%

  At the beginning of this year Structured Dynamics assembled a listing of ontology building tools at the request of a client. That listing was presented as The Sweet Compendium of Ontology Building Tools. Now, again because of some client and internal work, we have researched the space again and updated the listing [1].

  All new tools are marked with <New> (new only means newly discovered; some had yet to be discovered in the prior listing). There are now a total of 185 tools in the listing, 31 of which are recently new, and 45 added at various times since the first release.

  Comprehensive Ontology Tools

  • Altova SemanticWorks is a visual RDF and OWL editor that auto-generates RDF/XML or nTriples based on visual ontology design. No open source version available
  • Amine is a rather comprehensive, open source platform for the development of intelligent and multi-agent systems written in Java. As one of its components, it has an ontology GUI with text- and tree-based editing modes, with some graph visualization
  • The Apelon DTS (Distributed Terminology System) is an integrated set of open source components that provides comprehensive terminology services in distributed application environments. DTS supports national and international data standards, which are a necessary foundation for comparable and interoperable health information, as well as local vocabularies. Typical applications for DTS include clinical data entry, administrative review, problem-list and code-set management, guideline creation, decision support and information retrieval.. Though not strictly an ontology management system, Apelon DTS has plug-ins that provide visualization of concept graphs and related functionality that make it close to a complete solution
  • DOME is a programmable XML editor which is being used in a knowledge extraction role to transform Web pages into RDF, and available as Eclipse plug-ins. DOME stands for DERI Ontology Management Environment
  • FlexViz is a Flex-based, Protg-like client-side ontology creation, management and viewing tool; very impressive. The code is distributed from Sourceforge; there is a nice online demo available; there is a nice explanatory paper on the system, and the developer, Chris Callendar, has a useful blog with Flex development tips
  • Knoodl facilitates community-oriented development of OWL based ontologies and RDF knowledge bases. It also serves as a semantic technology platform, offering a Java service-based interface or a SPARQL-based interface so that communities can build their own semantic applications using their ontologies and knowledgebases. It is hosted in the Amazon EC2 cloud and is available for free; private versions may also be obtained. See especially the screencast for a quick introduction
  • The NeOn toolkit is a state-of-the-art, open source multi-platform ontology engineering environment, which provides comprehensive support for the ontology engineering life-cycle. The v2.3.0 toolkit is based on the Eclipse platform, a leading development environment, and provides an extensive set of plug-ins covering a variety of ontology engineering activities. You can add these plug-ins or get a current listing from the built-in updating mechanism
  • ontopia is a relative complete suite of tools for building, maintaining, and deploying Topic Maps-based applications; open source, and written in Java. Could not find online demos, but there are screenshots and there is visualization of topic relationships
  • Protg is a free, open source visual ontology editor and knowledge-base framework. The Protg platform supports two main ways of modeling ontologies via the Protg-Frames and Protg-OWL editors. Protg ontologies can be exported into a variety of formats including RDF(S), OWL, and XML Schema. There are a large number of third-party plugins that extends the platform's functionality
    • Protg Plugin Library frequently consult this page to review new additions to the Protg editor; presently there are dozens of specific plugins, most related to the semantic Web and most open source
    • Collaborative Protg is a plug-in extension of the existing Protg system that supports collaborative ontology editing as well as annotation of both ontology components and ontology changes. In addition to the common ontology editing operations, it enables annotation of both ontology components and ontology changes. It supports the searching and filtering of user annotations, also known as notes, based on different criteria. There is also an online demo
    • <New>Web Protg is an online version of Protg attempting to capture all of the native functionality; still under development
  • <New>Sigma is open source knowledge engineering environment that includes ontology mapping, theorem proving, language generation in multiple languages, browsing, OWL read/write, and analysis. It includes the Suggested Upper Merged Ontology (SUMO), a comprehensive formal ontology. It's under active development and use
  • TopBraid Composer is an enterprise-class modeling environment for developing Semantic Web ontologies and building semantic applications. Fully compliant with W3C standards, Composer offers comprehensive support for developing, managing and testing configurations of knowledge models and their instance knowledge bases. It is based on the Eclipse IDE. There is a free version (after registration) for small ontologies
  • <New>TwoUse Toolkit is an implementation of current OMG and W3C standards for developing ontology-based software models and model-based OWL2 ontologies, largely based around UML. There are a variety of tools, including graphics editors, with more to come
  • <New>Wandora is a topic maps engine written in Java with support for both in-memory topic maps and persisting topic maps in MySQL and SQL Server. It also contains an editor and a publishing system, and has support for automatic classification. It can read OBO, RDF(S), and many other formats, and can export topic maps to various graph formats. There is also a web-based topic maps browser, and graphical visualization.

  Not Apparently in Active Use

  • Adaptiva is a user-centred ontology building environment, based on using multiple strategies to construct an ontology, minimising user input by using adaptive information extraction
  • Exteca is an ontology-based technology written in Java for high-quality knowledge management and document categorisation, including entity extraction. Though code is still available, no updates have been provided since 2006. It can be used in conjunction with search engines
  • IODT is IBM's toolkit for ontology-driven development. The toolkit includes EMF Ontolgy Definition Metamodel (EODM), EODM workbench, and an OWL Ontology Repository (named Minerva)
  • KAON is an open-source ontology management infrastructure targeted for business applications. It includes a comprehensive tool suite allowing easy ontology creation and management and provides a framework for building ontology-based applications. An important focus of KAON is scalable and efficient reasoning with ontologies
  • Ontolingua provides a distributed collaborative environment to browse, create, edit, modify, and use ontologies. The server supports over 150 active users, some of whom have provided us with descriptions of their projects. Provided as an online service; software availability not known.

  Vocabulary Prompting Tools

  • AlchemyAPI from Orchestr8 provides an API based application that uses statistical and natural language processing methods. Applicable to webpages, text files and any input text in several languages
  • BooWa is a set expander for any language (formerly known as SEALS); developed by RC Wang of Carnegie Mellon
  • Google Keywords allows you to enter a few descriptive words or phrases or a site URL to generate keyword ideas
  • Google Sets for automatically creating sets of items from a few examples
  • Open Calais is free limited API web service to automatically attach semantic metadata to content, based on either entities (people, places, organizations, etc.), facts (person x' works for company y'), or events (person z' was appointed chairman of company y' on date x'). The metadata results are stored centrally and returned to you as industry-standard RDF constructs accompanied by a Globally Unique Identifier (GUID)
  • Query-by-document from BlogScope has a nice phrase extraction service, with a choice of ranking methods. Can also be used in a Firefox plug-in (not texted with 3.5+)
  • SemanticHacker (from Textwise) is an API that does a number of different things, including categorization, search, etc. By using concept tags', the API can be leveraged to generate metadata or tags for content
  • TagFinder is a Web service that automatically extracts tags from a piece of text. The tags are chosen based on both statistical and linguistic analysis of the original text
  • Tagthe.net has a demo and an API for automatic tagging of web documents and texts. Tags can be single words only. The tool also recognizes named entities such as people names and locations
  • TermExtractor extracts terminology consensually referred in a specific application domain. The software takes as input a corpus of domain documents, parses the documents, and extracts a list of syntactically plausible terms (e.g. compounds, adjective-nouns, etc.)
  • TermFinder uses Poisson statistics, the Maximum Likelihood Estimation and Inverse Document Frequency between the frequency of words in a given document and a generic corpus of 100 million words per language; available for English, French and Italian
  • TerMine is an online and batch term extractor that emphasizes part of speech (POS) and n-gram (phrase extraction). TerMine is the terminological management system with the C-Value term extraction and AcroMine acronym recognition integrated
  • Topia term extractor is a part-of-speech and frequency based term extraction tool implemented in python. Here is a term extraction demo based on this tool
  • Topicalizer is a service which automatically analyses a document specified by a URL or a plain text regarding its word, phrase and text structure. It provides a variety of useful information on a given text including the following: Word, sentence and paragraph count, collocations, syllable structure, lexical density, keywords, readability and a short abstract on what the given text is about
  • TrMExtractor does glossary extraction on pure text files for either English or Hungarian
  • Wikify! is a system to automatically wikify a text by adding Wikipedia-like tags throughout the document. The system extracts keywords and then disambiguates and matches them to their corresponding Wikipedia definition
  • Yahoo! Placemaker is a freely available geoparsing Web service. It helps developers make their applications location-aware by identifying places in unstructured and atomic content feeds, web pages, news, status updates and returning geographic metadata for geographic indexing and markup
  • Yahoo! Term Extraction Service is an API to Yahoo's term extraction service, as well as many other APIs and services in a variety of languages and for a variety of tasks; good general resource. The service has been reported to be shut down numerous times, but apparently is kept alive due to popular demand.

  Initial Ontology Development

  • COE COE (CmapTools Ontology Editor) is a specialized version of the CmapTools from IMHC. COE and its CmapTools parent is based on the idea of concept maps. A concept map is a graph diagram that shows the relationships among concepts. Concepts are connected with labeled arrows, with the relations manifesting in a downward-branching hierarchical structure. COE is an integrated suite of software tools for constructing, sharing and viewing OWL encoded ontologies based on these constructs
  • Conzilla2 is a second generation concept browser and knowledge management tool with many purposes. It can be used as a visual designer and manager of RDF classes and ontologies, since its native storage is in RDF. It also has an online collaboration server [apparently last updated in 2008]
  • http://diagramic.com/ has an online Flex network graph demo, which also has a neat facility for quick entry and visualization of relationships; mostly small scale; pretty cool. Does not appear to be code available anywhere
  • <New>DL-Learner is a tool for learning OWL class expressions from examples and background knowledge. It extends Inductive Logic Programming (ILP) to Description Logics and the Semantic Web. DL-Learner now has a flexible component based design, which allows to extend it easily with new learning algorithms, learning problems, reasoners, and supported background knowledge sources. A new type of supported knowledge sources are SPARQL endpoints, where DL-Learner can extract knowledge fragments, which enables learning classes even on large knowledge sources like DBpedia, and includes an OWL API reasoner interface and Web service interface.
  • DogmaModeler is a free and open source, ontology modeling tool based on ORM. The philosophy of DogmaModeler is to enable non-IT experts to model ontologies with a little or no involvement of an ontology engineer; project is quite old, but the software is still available and it may provide some insight into naive ontology development
  • Erca is a framework that eases the use of Formal and Relational Concept Analysis, a neat clustering technique. Though not strictly an ontology tool, Erca could be implemented in a work flow that allows easy import of formal contexts from CSV files, then algorithms that computes the concept lattice of the formal contexts that can be exported as dot graphs (or in JPG, PNG, EPS and SVG formats). Erca is provided as an Eclipse plug-in
  • GraphMind is a mindmap editor for Drupal. It has the basic mindmap features and some Drupal specific enhancements. There is a quick screencast about how GraphMind looks like and what is does. The Flex source is also available from Github
  • <New>H-Maps is a commercial suite of tools for building topic maps applications, consisting of a topic maps engine and server, a mapping framework for converting from legacy data, and a navigator for visualizing data. It is typically used in bioinformatics (drug discovery and research, toxicological studies, etc), engineering (support and expert systems), and for integration of hetereogeneous data. It supports the XTM 1.0 and TMAPI 1.0 specifications
  • irON using spreadsheets, via its notation and specification. Spreadsheets can be used for initial authoring, esp if the irON guidelines are followed. See further this case study of Sweet Tools in a spreadsheet using irON (commON)
  • ITM T3 stands for Terminology, Thesaurus, Taxonomy, Metadata dictionary. ITM T3 includes a range of functions for managing enterprise shareable multilingual domain-specific taxonomies, thesaurus, terminologies in a unified way. It uses XML, SKOS and RDF standards. Commercial; from Mondeca
  • <New>JXML2OWL API is a library for mapping XML schemas to OWL Ontologies on the JAVA platform. It creates an XSLT which transforms instances of the XML schema into instances of the OWL ontology. JXML2OWL Mapper is GUI application using the JXML2OWL API
  • MindRaider is Semantic Web outliner. It aims to connect the tradition of outline editors with emerging technologies. MindRaider mission is to organize not only the content of your hard drive but also your cognitive base and social relationships in a way that enables quick navigation, concise representation and inferencing
  • <New>Neologism is a simple web-based RDF Schema vocabulary editor and publishing system. Use it to create RDF classes and properties, which are needed to publish data on the Semantic Web. Its main goal is to dramatically reduce the time required to create, publish and modify vocabularies for the Semantic Web. It is written in PHP and built on the Drupal platform. Neologism is currently in alpha
  • <New>OCS Ontology Creation System is software to develop ontologies in cooperative way with a graphical interface
  • RDF123 is an application and web service for converting data in simple spreadsheets to an RDF graph. Users control how the spreadsheet's data is converted to RDF by constructing a graphical RDF123 template that specifies how each row in the spreadsheet is converted as well as metadata for the spreadsheet and its RDF translation
  • <New>ROC (Rapid Ontology Construction) is a tool that allows domain experts to quickly build a basic vocabulary for their domain, re-using existing terminology whenever possible. How this works is that the ROC tool asks the domain expert for a set of keywords that are core' terms of the domain, and then queries remote sources for concepts matching those terms. These are then presented to the user, who can select terms from the list, find relations to other terms, and expand the set of terms and relations, iteratively. The resulting vocabulary (or proto-ontology', basically a SKOS-like thesaurus) can be used as is, or can be used as input for a knowledge engineer to base a more comprehensive domain ontology on. Interface triples-oriented, not graphical.
  • Topincs is a Topic Map authoring software that allows groups to share their knowledge over the web. It makes use of a variety of modern technologies. The most important are Topic Maps, REST and Ajax. It consists of three components: the Wiki, the Editor, and the Server. The servier requires AMP; the Editor and Wiki are based on browser plug-ins.

  Ontology Editing

  • First, see all of the Comprehensive Tools and Ontology Development listings above
  • Anzo for Excel includes an (RDFS and OWL-based) ontology editor that can be used directly within Excel. In addition to that, Anzo for Excel includes the capability to automatically generate an ontology from existing spreadsheet data, which is very useful for quick bootstrapping of an ontology
  • <New>ATop is a topic map browser and editor written in Java and supports the XTM 1.0 specification; project has not been updated since 2008
  • Hozo is an ontology visualization and development tool that brings version control constructs to group ontology development; limited to a prototype, with no online demo
  • Lexaurus Editor is for off-line creation and editing of vocabularies, taxonomies and thesauri. It supports import and export in Zthes and SKOS XML formats, and allows hierarchical / poly-hierarchical structures to be loaded for editing, or even multiple vocabularies to be loaded simultaneously, so that terms from one taxonomy can be re-used in another, using drag and drop. Not available in open source
  • Model Futures OWL Editor combines simple OWL tools, featuring UML (XMI), ErWin, thesaurus and imports. The editor is tree-based and has a navigator tool for traversing property and class-instance relationships. It can import XMI (the interchange format for UML) and Thesaurus Descriptor (BT-NT XML), and EXPRESS XML files. It can export to MS Word.
  • <New>OBO-Edit is an open source ontology editor written in Java. OBO-Edit is optimized for the OBO biological ontology file format. It features an easy to use editing interface, a simple but fast reasoner, and powerful search capabilities
  • <New>Onotoa is an Eclipse-based ontology editor for topic maps. It has a graphical UML-like interface, an export function for the current TMCL-draft and a XTM export
  • OntoTrack is a browsing and editing ontology authoring tool for OWL Lite. It combines a sophisticated graphical layout with mouse enabled editing features optimized for efficient navigation and manipulation of large ontologies
  • OWLViz is an attractive visual editor for OWL and is available as a Protg plug-in
  • PoolParty is a triple store-based thesaurus management environment which uses SKOS and text extraction for tag recommendations. See further this manual, which describes more fully the system's functionality. Also, there is a PoolParty Web service that enables a Zthes thesaurus in XML format to be uploaded and converted to SKOS (via skos:Concepts)
  • SKOSEd is a plugin for Protege 4 that allows you to create and edit thesauri (or similar artefacts) represented in the Simple Knowledge Organisation System (SKOS).
  • TemaTres is a Web application to manage controlled vocabularies, taxonomies and thesaurus. The vocabularies may be exported in Zthes, Skos, TopicMap, etc.
  • ThManager is a tool for creating and visualizing SKOS RDF vocabularies. ThManager facilitates the management of thesauri and other types of controlled vocabularies, such as taxonomies or classification schemes
  • Vitro is a general-purpose web-based ontology and instance editor with customizable public browsing. Vitro is a Java web application that runs in a Tomcat servlet container. With Vitro, you can: 1) create or load ontologies in OWL format; 2) edit instances and relationships; 3) build a public web site to display your data; and 4) search your data with Lucene. Still in somewhat early phases, with no online demos and with minimal interfaces.
  • <New>Vocab Editor is an RDF/OWL/SKOS vocabulary-diagram editor. It has both client- (Javascript) and server-side (Python) implmentations. It is open source with a demo. There is a blog (Spanish) and online sample vocabulary app editor.

  Not Apparently in Active Use

  • Omnigator The Omnigator is a form-based manipulaton tool centered on Topic Maps, though it enables the loading and navigation of any conforming topic map in XTM, HyTM, LTM or RDF formats. There is a free evaluation version.
  • OntoGen is a semi-automatic and data-driven ontology editor focusing on editing of topic ontologies (a set of topics connected with different types of relations). The system combines text-mining techniques with an efficient user interface. It requires .Net.
  • OntoLight is a set of software modules for: transforming raw ontology data for several ontologies from their specific formats into a unifying light-weight ontology format, grounding the ontology and storing it into grounded ontology format, populating grounded ontologies with new instance data, and creating mappings between grounded ontologies; includes Cyc. Download no longer available. See http://analytics.ijs.si/~blazf/papers/Context_SiKDD07.pdf and http://www.neon-project.org/web-content/index.php?option=com_weblinks&task=view&catid=17&id=52 or http://www.neon-project.org/web-content/index.php?option=com_weblinks&catid=21&Itemid=73
  • OWL-S-editor is an editor for the development of services in OWL-S, with graphical, WSDL and import/export support
  • ReTAX+ is an aide to help a taxonomist create a consistent taxonomy and in particular provides suggestions as to where a new entity could be placed in the taxonomy whilst retaining the integrity of the revised taxonomy (c.f., problems in ontology modelling)
  • SWOOP is a lightweight ontology editor. (Swoop is no longer under active development at mindswap. Continuing development can be found on SWOOP's Google Code homepage at http://code.google.com/p/swoop/)
  • WebOnto supports the browsing, creation and editing of ontologies through coarse grained and fine grained visualizations and direct manipulation.

  Ontology Mapping

  • <New>The Alignment API is an API and implementation for expressing and sharing ontology alignments. The correspondences between entities (e.g., classes, objects, properties) in ontologies is called an alignment. The API provides a format for expressing alignments in a uniform way. The goal of this format is to be able to share on the web the available alignments. The format is expressed in RDF, so it is freely extensible. The Alignment API itself is a Java description of tools for accessing the common format. It defines four main interfaces (Alignment, Cell, Relation and Evaluator).
  • COMA++ is a schema and ontology matching tool with a comprehensive infrastructure. Its graphical interface supports a variety of interaction
  • ConcepTool is a system to model, analyse, verify, validate, share, combine, and reuse domain knowledge bases and ontologies, reasoning about their implication
  • <New>MapOnto is a research project aiming at discovering semantic mappings between different data models, e.g, database schemas, conceptual schemas, and ontologies. So far, it has developed tools for discovering semantic mappings between database schemas and ontologies as well as between different database schemas. The Protege plug-in is still available, but appears to be for older versions
  • MatchIT automates and facilitates schema matching and semantic mapping between different Web vocabularies. MatchIT runs as a stand-alone or plug-in Eclipse application and can be integrated with popular third party applications. MatchIT's uses Adaptive Lexicon as an ontology-driven dictionary and thesaurus of English language terminology to quantify and ank the semantic similarity of concepts. It apparently is not available in open source
  • myOntology is used to produce the theoretical foundations, and deployable technology for the Wiki-based, collaborative and community-driven development and maintenance of ontologies instance data and mappings
  • OLA/OLA2 (OWL-Lite Alignment) matches ontologies written in OWL. It relies on a similarity combining all the knowledge used in entity descriptions. It also deal with one-to-many relationships and circularity in entity descriptions through a fixpoint algorithm
  • Potluck is a Web-based user interface that lets casual usersthose without programming skills and data modeling expertisemash up data themselves. Potluck is novel in its use of drag and drop for merging fields, its integration and extension of the faceted browsing paradigm for focusing on subsets of data to align, and its application of simultaneous editing for cleaning up data syntactically. Potluck also lets the user construct rich visualizations of data in-place as the user aligns and cleans up the data.
  • PRIOR+ is a generic and automatic ontology mapping tool, based on propagation theory, information retrieval technique and artificial intelligence model. The approach utilizes both linguistic and structural information of ontologies, and measures the profile similarity and structure similarity of different elements of ontologies in a vector space model (VSM).
  • <New>S-Match takes any two tree like structures (such as database schemas, classifications, lightweight ontologies) and returns a set of correspondences between those tree nodes which semantically correspond to one another.
  • Vine is a tool that allows users to perform fast mappings of terms across ontologies. It performs smart searches, can search using regular expressions, requires a minimum number of clicks to perform mappings, can be plugged into arbitrary mapping framework, is non-intrusive with mappings stored in an external file, has export to text files, and adds metadata to any mapping. See also http://sourceforge.net/projects/vine/.

  Not Apparently in Active Use

  • ASMOV (Automated Semantic Mapping of Ontologies with Validation) is an automatic ontology matching tool which has been designed in order to facilitate the integration of heterogeneous systems, using their data source ontologies
  • Chimaera is a software system that supports users in creating and maintaining distributed ontologies on the web. Two major functions it supports are merging multiple ontologies together and diagnosing individual or multiple ontologies
  • CMS (CROSI Mapping System) is a structure matching system that capitalizes on the rich semantics of the OWL constructs found in source ontologies and on its modular architecture that allows the system to consult external linguistic resources
  • ConRef is a service discovery system which uses ontology mapping techniques to support different user vocabularies
  • DRAGO reasons across multiple distributed ontologies interrelated by pairwise semantic mappings, with a vision of peer-to-peer mapping of many distributed ontologies on the Web. It is implemented as an extension to an open source Pellet OWL Reasoner
  • Falcon-AO (Finding, aligning and learning ontologies) is an automatic ontology matching tool that includes the three elementary matchers of String, V-Doc and GMO. In addition, it integrates a partitioner PBM to cope with large-scale ontologies
  • FOAM is the Framework for ontology alignment and mapping. It is based on heuristics (similarity) of the individual entities (concepts, relations, and instances)
  • hMAFRA (Harmonize Mapping Framework) is a set of tools supporting semantic mapping definition and data reconciliation between ontologies. The targeted formats are XSD, RDFS and KAON
  • IF-Map is an Information Flow based ontology mapping method. It is based on the theoretical grounds of logic of distributed systems and provides an automated streamlined process for generating mappings between ontologies of the same domain
  • LILY is a system matching heterogeneous ontologies. LILY extracts a semantic subgraph for each entity, then it uses both linguistic and structural information in semantic subgraphs to generate initial alignments. The system is presently in a demo version only
  • MAFRA Toolkit the Ontology MApping FRAmework Toolkit allows users to create semantic relations between two (source and target) ontologies, and apply such relations in translating source ontology instances into target ontology instances
  • OntoEngine is a step toward allowing agents to communicate even though they use different formal languages (i.e., different ontologies). It translates data from a source ontology to a target
  • OWLS-MX is a hybrid semantic Web service matchmaker. OWLS-MX 1.0 utilizes both description logic reasoning, and token based IR similarity measures. It applies different filters to retrieve OWL-S services that are most relevant to a given query
  • RiMOM (Risk Minimization based Ontology Mapping) integrates different alignment strategies: edit-distance based strategy, vector-similarity based strategy, path-similarity based strategy, background-knowledge based strategy, and three similarity-propagation based strategies
  • semMF is a flexible framework for calculating semantic similarity between objects that are represented as arbitrary RDF graphs. The framework allows taxonomic and non-taxonomic concept matching techniques to be applied to selected object properties
  • Snoggle is a graphical, SWRL-based ontology mapper. Snoggle attempts to solve the ontology mapping problem by providing a graphical user interface (similar to which of the Microsoft Visio) to guide the process of ontology vocabulary alignment. In Snoggle, user-defined mappings can be serialized into rules, which is expressed using SWRL
  • Terminator is a tool for creating term to ontology resource mappings (documentation in Finnish).

  Ontology Visualization/Analysis

  Though all are not relevant, see my post from a couple of years back on large-scale RDF graph software.

  • Social network graphing tools (many covered elsewhere)
  • Cytoscape is a bioinformatics software platform for visualizing molecular interaction networks and integrating these interactions with gene expression profiles and other state data; I have also written specifically about Cytoscape's use in UMBEL
    • RDFScape is a project that brings Semantic Web features to the popular Systems Biology software Cytoscape
    • NetworkAnalyzer performs analysis of biological networks and calculates network topology parameters including the diameter of a network, the average number of neighbors, and the number of connected pairs of nodes. It also computes the distributions of more complex network parameters such as node degrees, average clustering coefficients, topological coefficients, and shortest path lengths. It displays the results in diagrams, which can be saved as images or text files; used by SD
  • Graphl is a tool for collaborative editing and visualisation of graphs, representing relationships between resources or concepts of the real world. Graphl may be thought of as a visual wiki, a place where everybody can contribute to a shared repository of knowledge
  • <New>Graphviz is open source graph visualization software. It has several main graph layout programs. It also has web and interactive graphical interfaces, and auxiliary tools, libraries, and language bindings.
  • <New>GrOWL is an ontology visualizer and editor. The layout of the GrOWL graph can be defined automatically or loaded from a separate style sheet. GrOWL implements configurable filters that can transform the display by simplifying it, hiding concepts and relationships that have no descriptions associated, or perform more complex translations. Concepts can be stored in ontologies with extensive annotations to provide documentation. GrOWL shows these annotation as tooltips, and supports complex HTML and links within them. The GrOWL browser can be used inside a web browser or as a stand-alone application. When used inside a browser, it supports Javascript interaction so that it can be used as a concept chooser with implementation-defined operations.
  • igraph is a free software package for creating and manipulating undirected and directed graphs
  • Network Workbench is a very complex, comprehensive; Swiss Army Knife
  • NetworkX Python; very clean
  • <New>OntoGraf, a Protege 4 plug-in, gives support for interactively navigating the relationships of your OWL ontologies. Various layouts are supported for automatically organizing the structure of your ontology. Different relationships are supported: subclass, individual, domain/range object properties, and equivalence. Relationships and node types can be filtered.
  • <New>OWL2Prefuse is a Java package which creats Prefuse graphs and trees from OWL files (and Jena OntModels). It takes care of converting the OWL data structure to the Prefuse datastructure. This makes it is easy for developers, to use the Prefuse graphs and trees into their Semantic Web applications.
  • <New>RDF Gravity is a tool for visualising RDF/OWL Graphs/ ontologies. RDF Gravity is implemented by using the JUNG Graph API and Jena semantic web toolkit. Its main features are:
    • Graph Visualization
    • Global and Local Filters (enabling specific views on a graph)
    • Full text Search
    • Generating views from RDQL Queries
    • Visualising multiple RDF files
  • Stanford Network Analysis Package (SNAP) is a general purpose network analysis and graph mining library. It is written in C++ and easily scales to massive networks with hundreds of millions of nodes
  • Social Networks Visualizer (SocNetV) is a flexible and user-friendly tool for the analysis and visualization of Social Networks. It lets you construct networks (mathematical graphs) with a few clicks on a virtual canvas or load networks of various formats (GraphViz, GraphML, Adjacency, Pajek, UCINET, etc) and modify them to suit your needs. SocNetV also offers a built-in web crawler, allowing you to automatically create networks from all links found in a given initial URL
  • Tulip may be incredibly strong
    • quite active (but not much online stuff): http://sourceforge.net/projects/auber/files/
  • Springgraph component for Flex
  • VizierFX is a Flex library for drawing network graphs. The graphs are laid out using GraphViz on the server side, then passed to VizierFX to perform the rendering. The library also provides the ability to run ActionScript code in response to events on the graph, such as mousing over a node or clicking on it.
  • <New>VUE (Visual Understanding Environment) is an open source project focused on creating flexible tools for managing and integrating digital resources in support of teaching, learning and research. VUE provides a flexible visual environment for structuring, presenting, and sharing digital information.
  • <New>yEd is a diagram editor that can be used to quickly and effectively generate high-quality drawings of diagrams. It can support OWL imports.
  • <New>ZGRViewer is a graph visualizer implemented in Java and based upon the Zoomable Visual Transformation Machine. It is specifically aimed at displaying graphs expressed using the DOT language from AT&T GraphViz and processed by programs dot, neato or others such as twopi. ZGRViewer is designed to handle large graphs, and offers a zoomable user interface (ZUI), which enables smooth zooming and easy navigation in the visualized structure.

  Miscellaneous Ontology Tools

  • Apolda (Automated Processing of Ontologies with Lexical Denotations for Annotation) is a plugin (processing resource) for GATE (http://gate.ac.uk/). The Apolda processing resource (PR) annotates a document like a gazetteer, but takes the terms from an (OWL) ontology rather than from a list
  • <New>Gloze is a XML to RDF, RDF to XML, and XSD to OWL mapping tool based on Jena; see also http://jena.hpl.hp.com/juc2006/proceedings/battle/paper.pdf . See also http://jena.sourceforge.net/contrib/contributions.html
  • <New>Hoolet is an implementation of an OWL-DL reasoner that uses a first order prover. The ontology is translated to collection of axioms (in an obvious way based on the OWL semantics) and this collection of axioms is then given to a first order prover for consistency checking.
  • LexiLink is a tool for building, curating and managing multiple lexicons and ontologies in one enterprise-wide Web-based application. The core of the technology is based on RDF and OWL
  • ontology owl ontologies based web ontology ontologies based tool semantic data editor system source knowledge mapping tools protg provides text development open java used service topic interface user software graph language supports different plug maps written online graphs domain applications information application browser terms extraction including mappings library relationships framework automatically support skos management create enables allows files uses visual visualization project format graphical comprehensive platform vocabularies vocabulary analysis multiple formats model toolkit concepts includes similarity environment network concept listing simple thesaurus building server given search large active python classes relations sources alignment matching distributed structure demo variety content term schema networks export part tags learning properties metadata creating document easy apparently classification systems code creation free version terminology main reasoner features modeling instances schemas engineering automatic eclipse flex implementation linguistic extracts collaborative several jena resources managing components users prolog flexible designed client module implemented growl complex component generating import integrated html requires bases taxonomies taxonomy swoop tree protege spreadsheet external transform entity entities constructs quick generate google allowing plugin developed browsing languages english instance class offers formal documents obda design alignments models runs annotation manipulation expressed repository driven perform strategy defined graphviz input target extension first node addition provided database representation provide hierarchical piece supported current list versions initial sourceforge build takes developers resource standards suite social processing navigation technology modify descriptions editors keywords converting drupal specification words services wiki open source semantic web topic maps web service web based ontology mapping owl ontology owl ontologies ontology editor based ontology web ontology ontology based knowledge bases source ontology plug ins term extraction ontology development rdf xml ontology building user interface mapping framework owl api text files ontology engineering database schemas based strategy ontology management ontology browser ontology language multiple ontologies distributed ontologies topic map semantic mappings semantic mapping visual ontology extraction service rdf schema background knowledge tool based knowledge sources automatic ontology graph visualization sourceforge net language owl owl lite ontology creation online demo ontology matching data sources xml schema building tools music ontology owl files web toolkit matching tool neon project web content content index instance data index php discovering semantic online tool grounded ontologies ontology manager objects properties semantic similarity stand alone vocabulary importer alignment api external vocabulary different data entity descriptions net projects rdf gravity main features web applications prefuse graphs used inside web browser network analysis social networks first order order prover processing resource user friendly obda ontology object properties similarity based data source source ontologies managing multiple allows users structural information swi prolog owl reasoner ontology instances ontology format based tool large scale given ontology mapping tool excel includes formats including third party ontology components modeling ontologies online demos life cycle based applications ontology changes under active publishing system knowledge management maps engine w3c standards active development web ontologies comprehensive support engineering environment provides comprehensive information retrieval apelon dts tree based ontology tools owl editor management system web pages based interface semantic applications based ontologies owl based eclipse plug management environment ontology driven driven development simple knowledge web application protg plug ontology file ontology visualization vocabularies taxonomies controlled vocabularies general purpose topic ontologies ontology data server side diagram editor web site rdf owl editor written rdf graph linguistic analysis application domain phrase extraction carnegie mellon ontology repository based application term extractor given text jxml2owl api graphical interface formal contexts component based management tool rdf classes several ontologies web ontology language ontology building tools open source ontology ontology matching tool similarity based strategy based ontology mapping first order prover web based tool external vocabulary importer under active development automatic ontology matching mapping tool based web based ontology based ontology editor web content index topic maps engine discovering semantic mappings term extraction service ]]> Mon, 23 Aug 2010 07:25:19 -0400 30:00 tag:filome.com,3 http://feedproxy.google.com/~r/readwriteweb/~3/Xfru81DhuwE/membase-the-database-powering.php Shared item ReadWriteWeb
    First shared by - robdiana
    syndication+ 0 | Search 1 | Shares 1
    
    

    65 million people play Zynga's online games every day. Millions of web browsers open to millions farms and millions of frontiers. They take turns, they tend crops, they send gifts. They buy millions of objects and upgrades. It's a mind-boggling amount of data. It's a new sort of data, and it warranted development of a new sort of database management system.

    That system, Membase, is a "NoSQL" database optimized for storing web applications. Membase was developed by Zynga, NorthScale, and NHN, and its source code released as open source in June of this year.

    Sponsor
    

    RDBMS Redux

    Membase is one of a number of new databases that break from the relational database management system (RDBMS) model. The RDBMS has a long history, dating back to the 1970s. In a relational database, data is stored in the form of tables, as is the relationship among the data. This system has worked well to handle transactional and structured data.

    But as the amount of information, the kind of information, the number of users accessing the information have grown, the relational database has faced some challenges. With new data comes new storage demands. And the traditional RDBMS is not optimized for the kind of environment that big data and cloud computing have created - one that's elastic and distributed.

    From Cache to Database

    Memcached is a tool that was developed to help address some of the problems as our computing needs shifted. Originally built by Brad Fitzpatrick for LiveJournal in 2003, memcached is distributed memory caching system. But according to James Phillips, Senior VP of Products at NorthScale, many applications have been using memcached for more than just transient storage. "People like memcached because it represents a practically boundless place to easily cache data, at very low cost and with predictably stellar performance. No schemas, no tables, no sharding, no normalizing, no tuning."

    NorthScale was founded by some of the leaders in the open source memcached community, and the company, along with Zynga, has taken that expertise to develop Membase, so that the same speed, flexibility, and simplicity of memcached could made to really store data in a database, not just a cache.

    Phillips says that memcached is designed to dock next to the relational database management system, and it's already seen as a "best practice bandaid." So from there, NorthScale argues that it can help move its customers from existing RDBMS technology towards a database architecture that's more scalable - in other words, to Membase.

    It helps, of course, if you're going to argue for a move away from the RDBMS to have an early success story like Zynga to point to: the 500,000 ops-per-second database behind Farmville.

    Discuss
    
    
    
    database data memcached rdbms membase database data memcached rdbms system membase millions relational zynga northscale relational database management system database management open source database management system relational database management ]]>ReadWriteWeb
    First shared by - robdiana
    syndication+ 0 | Search 1 | Shares 1
    
    

    65 million people play Zynga's online games every day. Millions of web browsers open to millions farms and millions of frontiers. They take turns, they tend crops, they send gifts. They buy millions of objects and upgrades. It's a mind-boggling amount of data. It's a new sort of data, and it warranted development of a new sort of database management system.

    That system, Membase, is a "NoSQL" database optimized for storing web applications. Membase was developed by Zynga, NorthScale, and NHN, and its source code released as open source in June of this year.

    Sponsor
    

    RDBMS Redux

    Membase is one of a number of new databases that break from the relational database management system (RDBMS) model. The RDBMS has a long history, dating back to the 1970s. In a relational database, data is stored in the form of tables, as is the relationship among the data. This system has worked well to handle transactional and structured data.

    But as the amount of information, the kind of information, the number of users accessing the information have grown, the relational database has faced some challenges. With new data comes new storage demands. And the traditional RDBMS is not optimized for the kind of environment that big data and cloud computing have created - one that's elastic and distributed.

    From Cache to Database

    Memcached is a tool that was developed to help address some of the problems as our computing needs shifted. Originally built by Brad Fitzpatrick for LiveJournal in 2003, memcached is distributed memory caching system. But according to James Phillips, Senior VP of Products at NorthScale, many applications have been using memcached for more than just transient storage. "People like memcached because it represents a practically boundless place to easily cache data, at very low cost and with predictably stellar performance. No schemas, no tables, no sharding, no normalizing, no tuning."

    NorthScale was founded by some of the leaders in the open source memcached community, and the company, along with Zynga, has taken that expertise to develop Membase, so that the same speed, flexibility, and simplicity of memcached could made to really store data in a database, not just a cache.

    Phillips says that memcached is designed to dock next to the relational database management system, and it's already seen as a "best practice bandaid." So from there, NorthScale argues that it can help move its customers from existing RDBMS technology towards a database architecture that's more scalable - in other words, to Membase.

    It helps, of course, if you're going to argue for a move away from the RDBMS to have an early success story like Zynga to point to: the 500,000 ops-per-second database behind Farmville.

    Discuss
    
    
    
    database data memcached rdbms membase database data memcached rdbms system membase millions relational zynga northscale relational database management system database management open source database management system relational database management ]]> Wed, 18 Aug 2010 18:40:37 -0400 30:00 tag:filome.com,4 http://blogs.oracle.com/optimizer/2008/02/displaying_and_reading_the_execution_plans_for_a_sql_statement.html Shared item Inside the Oracle Optimizer
    First shared by - robdiana
    syndication+ 0 | Search 1 | Shares 1
    
    

    Shared by robdiana
    If you use Oracle and your query is a bit slow, read this. Please read this. Your DBA's will love you for it.

    Generating and displaying the execution plan of a SQL statement is a common task for most DBAs, SQL developers, and preformance experts as it provides them information on the performance characteristics of a SQL statement. An execution plan shows the detailed steps necessary to execute a SQL statement. These steps are expressed as a set of database operators that consumes and produces rows. The order of the operators and their implentation is decided by the query optimizer using a combination of query transformations and physical optimization techniques.
    
    While the display is commonly shown in a tabular format, the plan is in fact tree-shaped. For example, consider the following query based on the SH schema (Sales History):
    
    
    

    
    select prod_category, avg(amount_sold)
    from sales s, products p
    where p.prod_id = s.prod_id
    group by prod_category;

    
    
    The tabular representation of this query's plan is:
    
    
    

    
    ------------------------------------------
     Id   Operation              Name   
    ------------------------------------------
       0  SELECT STATEMENT              
       1   HASH GROUP BY                
       2    HASH JOIN                   
       3     TABLE ACCESS FULL   PRODUCTS
       4     PARTITION RANGE ALL        
       5      TABLE ACCESS FULL  SALES  
    ------------------------------------------
    

    
    While the tree-shaped representation of the plan is:
    

    
       GROUP BY
          |
         JOIN
     _____|_______
     |            |
    ACCESS     ACCESS
    (PRODUCTS) (SALES)
    

    
    

    
    
    
    When you read a plan tree you should start from the bottom up. In the above example begin by looking at the access operators (or the leaves of the tree). In this case the access operators are implemented using full table scans. The rows produced by these tables scans will be consumed by the join operator. Here the join operator is a hash-join (other alternatives include nested-loop or sort-merge join). Finally the group-by operator implemented here using hash (alternative would be sort) consumes rows produced by the join-opertor.
    
    The execution plan generated for a SQL statement is just one of the many alternative execution plans considered by the query optimizer. The query optimizer selects the execution plan with the lowest cost. Cost is a proxy for performance, the lower is the cost the better is the performance. The cost model used by the query optimizer accounts for the IO, CPU, and network usage in the query.
    
    There are two different methods you can use to look at the execution plan of a SQL statement:
    

    
    
    1. EXPLAIN PLAN command - This displays an execution plan for a SQL statement without actually executing the statement.
    
    
    2. V$SQL_PLAN - A dictionary view introduced in Oracle 9i that shows the execution plan for a SQL statement that has been compiled into a cursor in the cursor cache.
    
    
    
    Under certain conditions the plan shown when using EXPLAIN PLAN can be different from the plan shown using V$SQL_PLAN. For example, when the SQL statement contains bind variables the plan shown from using EXPLAIN PLAN ignores the bind variable values while the plan shown in V$SQL_PLAN takes the bind variable values into account in the plan generation process.
    
    Displaying an execution plan has been made easier after the introduction of the dbms_xplan package in Oracle 9i and by the enhancements made to it in subsequent releases. This packages provides several PL/SQL procedures to display the plan from different sources:
    
    
    
    
    1. EXPLAIN PLAN command
    
    
    2. V$SQL_PLAN
    
    
    3. Automatic Workload Repository (AWR)
    
    
    4. SQL Tuning Set (STS)
    
    
    5. SQL Plan Baseline (SPM)
    
    
    
    The following examples illustrate how to generate and display an execution plan for our original SQL statement using the different functions provided in the dbms_xplan package.
    
    Example 1 Uses the EXPLAIN PLAN command and the dbms_xplan.display function.
    
    

    
    SQL> EXPLAIN PLAN FOR
     2   select prod_category, avg(amount_sold)
     3   from sales s, products p
     4   where p.prod_id = s.prod_id
     5   group by prod_category;
    
    Explained.
    

    
    
    

    
    SQL> select plan_table_output
     2    from table(dbms_xplan.display('plan_table',null,'basic'));
    
    ------------------------------------------
     Id   Operation              Name   
    ------------------------------------------
       0  SELECT STATEMENT              
       1   HASH GROUP BY                
       2    HASH JOIN                   
       3     TABLE ACCESS FULL   PRODUCTS
       4     PARTITION RANGE ALL        
       5      TABLE ACCESS FULL  SALES  
    ------------------------------------------
    

    
    
    The arguments are for dbms_xplan.display are:
    
    
    
    
    • plan table name (default 'PLAN_TABLE'),
    
    
    • statement_id (default null),
    
    
    • format (default 'TYPICAL')
    
    
    
    More details can be found in $ORACLE_HOME/rdbms/admin/dbmsxpln.sql.
    
    Example 2 Generating and displaying the execution plan for the last SQL statement executed in a session:
    
    
    

    
    SQL> select prod_category, avg(amount_sold)
     2   from sales s, products p
     3   where p.prod_id = s.prod_id
     4   group by prod_category;
    
    no rows selected
    

    
    

    
    SQL> select plan_table_output
     2    from table(dbms_xplan.display_cursor(null,null,'basic'));
    
    ------------------------------------------
     Id   Operation              Name   
    ------------------------------------------
       0  SELECT STATEMENT              
       1   HASH GROUP BY                
       2    HASH JOIN                   
       3     TABLE ACCESS FULL   PRODUCTS
       4     PARTITION RANGE ALL        
       5      TABLE ACCESS FULL  SALES  
    ------------------------------------------
    

    
    
    The arguments used by dbms_xplay.dispay_cursor are:
    
    
    
    
    • SQL ID (default null, null means the last SQL statement executed in this session),
    
    
    • child number (default 0),
    
    
    • format (default 'TYPICAL')
    
    
    
    The details are in $ORACLE_HOME/rdbms/admin/dbmsxpln.sql.
    
    
    Example 3 Displaying the execution plan for any other statement requires the SQL ID to be provided, either directly or indirectly:
    
    
    1. Directly:

       
       SQL> select plan_table_output from
        2   table(dbms_xplan.display_cursor('fnrtqw9c233tt',null,'basic'));

       
       
       
    2. Indirectly:

       
       SQL> select plan_table_output
        2   from v$sql s,
        3   table(dbms_xplan.display_cursor(s.sql_id,
        4                                  s.child_number, 'basic')) t
        5   where s.sql_text like 'select PROD_CATEGORY%';

       
       
    
    Example 4 - Displaying an execution plan corresponding to a SQL Plan Baseline. SQL Plan Baselines have been introduced in Oracle 11g to support the SQL Plan Management feature (SPM). In order to illustrate such a case we need to create a SQL Plan Baseline first.
    
    

    
    SQL> alter session set optimizer_capture_sql_plan_baselines=true;
    
    Session altered.
    
    SQL> select prod_category, avg(amount_sold)
     2   from sales s, products p
     3   where p.prod_id = s.prod_id
     4   group by prod_category;
    
    no rows selected
    

    
    If the above statement has been executed more than once, a SQL Plan Baseline will be created for it and you can verified this using the follows query:
    
    

    
    SQL> select SQL_HANDLE, PLAN_NAME, ACCEPTED
     2   from dba_sql_plan_baselines
     3   where sql_text like 'select prod_category%';
    
    SQL_HANDLE                     PLAN_NAME                      ACC
    ------------------------------ ------------------------------ ---
    SYS_SQL_1899bb9331ed7772       SYS_SQL_PLAN_31ed7772f2c7a4c2  YES
    

    
    
    The execution plan for the SQL Plan Baseline created above can be displayed either directly or indirectly:
    
    
    1. Directly
       

       select t.* from
       table(dbms_xplan.display_sql_plan_baseline('SYS_SQL_1899bb9331ed7772',
                                                  format => 'basic')) t

       
       
       
    2. Indirectly
       

       select t.*
           from (select distinct sql_handle
                 from dba_sql_plan_baselines
                 where sql_text like 'select prod_category%') pb,
                table(dbms_xplan.display_sql_plan_baseline(pb.sql_handle,
                                                           null,'basic')) t;

       
       
    
    
    The output of either of these two statements is:
    
    
    

    
    ----------------------------------------------------------------------------
    SQL handle: SYS_SQL_1899bb9331ed7772
    SQL text: select prod_category, avg(amount_sold) from sales s, products p
              where p.prod_id = s.prod_id group by prod_category
    ----------------------------------------------------------------------------
    
    ----------------------------------------------------------------------------
    Plan name: SYS_SQL_PLAN_31ed7772f2c7a4c2
    Enabled: YES     Fixed: NO      Accepted: YES     Origin: AUTO-CAPTURE
    ----------------------------------------------------------------------------
    
    Plan hash value: 4073170114
    
    ---------------------------------------------------------
     Id   Operation                 Name               
    ---------------------------------------------------------
       0  SELECT STATEMENT                             
       1   HASH GROUP BY                               
       2    HASH JOIN                                  
       3     VIEW                   index$_join$_002   
       4      HASH JOIN                                
       5       INDEX FAST FULL SCAN PRODUCTS_PK        
       6       INDEX FAST FULL SCAN PRODUCTS_PROD_CAT_IX
       7     PARTITION RANGE ALL                       
       8      TABLE ACCESS FULL     SALES              
    ---------------------------------------------------------
    

    
    
    

    Formatting

    
    The format argument is highly customizable and allows you to see as little (high-level) or as much (low-level) details as you need / want in the plan output. The high-level options are:
    
    
    1. Basic
       The plan includes the operation, options, and the object name (table, index, MV, etc)
       
    2. Typical
       It includes the information shown in BASIC plus additional optimizer-related internal information such as cost, size, cardinality, etc. These information are shown for every operation in the plan and represents what the optimizer thinks is the operation cost, the number of rows produced, etc. It also shows the predicates evaluation by the operation. There are two types of predicates: ACCESS and FILTER. The ACCESS predicates for an index are used to fetch the relevant blocks because they apply to the search columns. The FILTER predicates are evaluated after the blocks have been fetched.
       
    3. All
       It includes the information shown in TYPICAL plus the lists of expressions (columns) produced by every operation, the hint alias and query block names where the operation belongs. The last two pieces of information can be used as arguments to add hints to the statement.
       
    The low-level options allow the inclusion or exclusion of find details, such as predicates and cost.
    For example,
    
    
    

    select plan_table_output
    from table(dbms_xplan.display('plan_table',null,'basic +predicate +cost'));
    
    -------------------------------------------------------
     Id   Operation              Name      Cost (%CPU)
    -------------------------------------------------------
       0  SELECT STATEMENT                    17  (18)
       1   HASH GROUP BY                      17  (18)
    *  2    HASH JOIN                         15   (7)
       3     TABLE ACCESS FULL   PRODUCTS      9   (0)
       4     PARTITION RANGE ALL               5   (0)
       5      TABLE ACCESS FULL  SALES         5   (0)
    -------------------------------------------------------
    
    Predicate Information (identified by operation id):
    ---------------------------------------------------
    2 - access("P"."PROD_ID"="S"."PROD_ID")
    

    
    
    
    

    select plan_table_output from
    table(dbms_xplan.display('plan_table',null,'typical -cost -bytes'));
    
    ----------------------------------------------------------------------------
     Id   Operation              Name      Rows  Time      Pstart Pstop
    ----------------------------------------------------------------------------
       0  SELECT STATEMENT                    4  00:00:01             
       1   HASH GROUP BY                      4  00:00:01             
    *  2    HASH JOIN                       960  00:00:01             
       3     TABLE ACCESS FULL   PRODUCTS   766  00:00:01             
       4     PARTITION RANGE ALL            960  00:00:01      1     16
       5      TABLE ACCESS FULL  SALES      960  00:00:01      1     16
    ----------------------------------------------------------------------------
    
    Predicate Information (identified by operation id):
    ---------------------------------------------------
    2 - access("P"."PROD_ID"="S"."PROD_ID")
    

    
    

    Note Section

    
    In addition to the plan, the package displays notes in the NOTE section, such as that dynamic sampling was used during query optimization or that star transformation was applied to the query.
    For example, if the table SALES did not have statistics then the optimizer will use dynamic sampling and the plan display will report it as follows (see '+note' detail in the query):
    
    
    

    select plan_table_output
    from table(dbms_xplan.display('plan_table',null,'basic +note'));
    
    ------------------------------------------
     Id   Operation              Name   
    ------------------------------------------
       0  SELECT STATEMENT              
       1   HASH GROUP BY                
       2    HASH JOIN                   
       3     TABLE ACCESS FULL   PRODUCTS
       4     PARTITION RANGE ALL        
       5      TABLE ACCESS FULL  SALES  
    ------------------------------------------
    
    Note
    -----
    - dynamic sampling used for this statement
    

    
    
    

    Bind peeking

    
    
    The query optimizer takes into account the values of bind variable values when generation an execution plan. It does what is generally called bind peeking. See the first post in this blog about the concept of bind peeking and its impact on the plans and the performance of SQL statements.
    As stated earlier the plan shown in V$SQL_PLAN takes into account the values of bind variables while the one shown from using EXPLAIN PLAN does not. Starting with 10gR2, the dbms_xplan package allows the display of the bind variable values used to generate a particular cursor/plan. This is done by adding '+peeked_binds' to the format argument when using display_cursor().
    This is illustrated with the following example:
    
    
    

    
    variable pcat varchar2(50)
    exec :pcat := 'Women'
    
    select PROD_CATEGORY, avg(amount_sold)
    from sales s, products p
    where p.PROD_ID = s.PROD_ID
    and prod_category != :pcat
    group by PROD_CATEGORY;
    
    select plan_table_output
    from table(dbms_xplan.display_cursor(null,null,'basic +PEEKED_BINDS'));
    
    ------------------------------------------
     Id   Operation              Name   
    ------------------------------------------
       0  SELECT STATEMENT              
       1   HASH GROUP BY                
       2    HASH JOIN                   
       3     TABLE ACCESS FULL   PRODUCTS
       4     PARTITION RANGE ALL        
       5      TABLE ACCESS FULL  SALES  
    ------------------------------------------
    
    Peeked Binds (identified by position):
    --------------------------------------
    
    1 - :PCAT (VARCHAR2(30), CSID=2): 'Women'

    
    
    plan sql table prod id plan table prod select statement access hash display sales products join dbms execution operation query xplan group name category null shown example basic cost bind cursor optimizer partition information range output rows used baseline default values displaying format oracle index predicates handle variable text baselines level session produced tree operators different package typical details sql plan table access dbms xplan plan table execution plan prod category xplan display sql statement hash join select prod table dbms partition range hash group select statement