Author Archive

ChemSpider data cleanup

15 Dec 2023

In previous posts, we have discussed the automated workflow we use to check new incoming data for structure and synonym errors. These checks allow us to remove the most common types of errors before they are added to the site. However, these filters do not apply to data already in ChemSpider.

Manual curation is an important part of our work. We periodically review the data on our most accessed records, in addition to ad-hoc removal or correction of erroneous data that we or our users notice when using the site. However, there are far too many records and far too much data to clean up using manual curation alone.

Recently we have focused on bulk identification and removal of erroneous data. This work has covered mapping errors and other clearly incorrect values in our experimental property data, correction or removal of malformed synonyms, correction of incorrectly labelled synonyms, and resolution of structure/synonym clashes.

Experimental Properties

We retrieved all 6.3 million experimental properties, text properties, and associated annotations from the ChemSpider database. We then compared the original text of the property as it was written in the original file to how that text was parsed and mapped by our deposition system. This enabled us to identify and correct several types of errors affecting around 2% of the properties in our database:

35,774 experimental property values had been assigned the incorrect unit (e.g. g/L instead of g/mL, °C instead of °F)
2,591 boiling points measured under non-standard pressure did not have this pressure displayed
4,292 densities had their density and temperature values swapped
79,252 miscellaneous erroneous properties and associated annotations were deleted. For example, “white crystals” mapped as melting point, impossibly high melting points or densities, etc.

Synonyms

Synonyms, chemical names, and identifiers are the most abundant type of data on ChemSpider, with a total of more than 446 million synonyms. These synonyms have additional metadata including language labels and flags identifying what type of synonym they are (e.g. CAS number, UNII, INN, trade name).

Simple Checks

We ran a series of regular expression string searches to identify synonyms with incorrect metadata, as well as malformed or otherwise erroneous synonyms.

200,007 synonym type flags added, and 4,766 incorrect flags removed
9,170 synonyms with an incorrect language label identified.
631,697 erroneous synonyms identified, including scrambled characters, properties/units, molecular formulae as synonyms, purity information, or invalid CAS numbers or EC numbers (formerly called EINECS).
922,334 instances of these erroneous synonyms deleted from ChemSpider records.

Structure/Synonym comparison

After identifying and removing these synonym-level errors, we then cross-checked ChemSpider records and their synonyms to identify mismatches. This work included amino acids, nucleic acids, and pharmaceutically acceptable salts.

As a first pass, we compared synonyms to molecular formulae to identify records missing key elements. Examples include synonyms describing a sodium salt when the molecular formula does not contain sodium, or describing an amino acid when the molecular formula contains no nitrogen. A total of 28,194 of these synonym/formula clashes were identified and removed.

For records that passed this initial molecular formula check, we performed a SMARTS comparison to identify chemical structures missing key structural features described in the synonym. These SMARTS strings were written broadly, with common substitutions allowed to prevent unnecessary removal of valid synonyms from derivative compounds.

In the following examples, the mismatched part of the synonym is highlighted in bold.

Structure	Removed synonym
	Sulfate ion
	Zolpidem tartrate
	Sodium S-sulfocysteine hydrate

After identifying these clashes, we manually spot-checked the output to weed out false positives and iterate the SMARTS filters. 101,257 synonym/structure clashes were identified and removed.

These checks included the following categories:

Amino acids and their derivatives: 6 formula clashes, 56 structure clashes
Nucleic acids, nucleosides, nucleotides: 977 formula clashes, 1,870 structure clashes
Halogens: 13,437 formula clashes, 1,256 structure clashes
Alkali and alkaline earth metals, and aluminium: 3,586 formula clashes, 56 structure clashes
Carboxylic acids and their derivatives: 5,002 formula clashes, 88,501 structure clashes
Other pharmaceutically acceptable acids: 3,534 formula clashes, 1,529 structure clashes
Amides and amines: 190 formula clashes, 304 structure clashes
Deuterates, hydrates, methylbromides: 1,462 formula clashes, 7,685 structure clashes

Get involved

You are the expert in your area of chemistry, so if you see something that doesn’t look quite right please let us know. If the error is confined to a single ChemSpider record, click the “Comment On This Record” box at the top of the affected record and let us know what the problem is. All we need is a sentence describing the error, however the more information you can provide, the better.

For more systemic errors, or in cases where you want to attach supplementary information or corrected chemical structures, please get in touch via email (chemspider@rsc.org).

Comments Off on ChemSpider data cleanup

Tips and tricks: generating machine-readable structural data from a structure

08 Nov 2019

Interested in making your article more discoverable and usable? As a reader, you have probably spent a lot of time re-drawing structures from an image in a PDF, or have struggled to find all relevant articles because your compound of interest is called by different names in different articles (IUPAC name, trivial name, registry number, drug development ID, generic name, brand name, revised trivial name etc etc etc…).

If you’re already drawing a structure for an article you are preparing to submit, it only takes a few seconds to generate machine-readable mol files or structure identifiers like SMILES or InChI. Including these files or identifiers in your article or supplementary information helps make your article indexable and structure-searchable, and is a great way to make your article stand out.

Save as MOL file

All major structure drawing packages can save structures as MOL files. They generally follow the same steps:

Choose File > Save As from the top menu OR press Ctrl+Shift+S.

Select “MDL Molfile”, “MDL SDFile”, or “.mol” or “.sdf” in the dropdown.

Please note: There may be more than one molfile format listed in the dropdown. If there is more than one option, please be aware that V2000 mol format is more common and is supported by all cheminformatics software packages. The V3000 mol file has some extra features, but is not universally supported, so it is advised that you use V2000 mol format to ensure maximum interoperability.

Copy as SMILES or InChI

Start by selecting the structure you would like to copy as SMILES or InChI.

Avogadro

From the top menu, choose Edit > Copy As > SMILES or InChI

ChemDoodle

From the top menu, choose Edit > Copy As > Daylight SMILES or IUPAC InChI

To copy as SMILES, press Ctrl+Alt+C

ChemDraw

From the top menu, choose Edit > Copy As > SMILES or InChI

Right click, and choose Molecule > Copy As > SMILES or InChI

To copy as SMILES, press Alt+Ctrl+C

ChemSketch

From the top menu, choose Tools > Generate > SMILES Notation or InChI for Structure

MarvinSketch

Press Ctrl+K, then select SMILES or InChI from the Copy As pop-up

From the top menu, choose Edit > Copy As and select SMILES or InChI from the pop-up

To copy as SMILES, press Ctrl+L

Finally, paste your SMILES or InChI into your document or spreadsheet.

The less time we have to spend re-drawing structures from pdfs, the more time we can devote to doing science. Luckily, it really couldn’t be quicker or easier to improve the discoverability and reusability of your article by including machine-readable structure files or identifiers. Let’s work together to make chemistry articles easier to find and use.

Comments Off on Tips and tricks: generating machine-readable structural data from a structure

ChemSpider Mobile app

28 Oct 2019

ChemSpider Mobile was an app developed by Molecular Materials Informatics Inc¹ on behalf of the Royal Society of Chemistry to allow users to explore the benefits of ChemSpider on mobile devices. Since its launch we have made improvements to ChemSpider.com, including responsive design elements to allow it to work better on smart phones and tablets² and upgrades to the ChemSpider web services³ that power it. As a result of these developments we felt it was timely to review the community’s need for the app and have taken the decision to discontinue support for the services that power the app from 31^st October. We would like to thank everyone who used and provided feedback on the app to aid its development and encourage you to switch to using ChemSpider.com for future mobile use.

1. http://molmatinf.com/

2. http://blogs.rsc.org/chemspider/2015/05/21/introduction-to-the-new-chemspider-website/

3. https://developer.rsc.org/

Comments Off on ChemSpider Mobile app

ChemSpider Pre-Deposition Filters

18 Sep 2018

Written by Mark Archibald.

In a previous post (Behind the Scenes at ChemSpider) we discussed some of the challenges in upholding data quality across one of the largest chemical databases in the world. We identified automated filtering as a key tool when dealing with far more records than a human could reasonably handle. In this post we’ll go into more detail about how that filtering works, what the challenges are, and the role played by human intervention.

To perform this filtering we use KNIME, an open-source data processing platform. The wide range of KNIME nodes developed by the active cheminformatics community allows us to ask chemistry-specific questions of the data we process. In simple terms, input chemical structures that match our criteria are passed on to the next node, while those that don’t are written out to an error file. After processing all structures, the result is a file of structures that have successfully passed through all the filters and several (usually smaller) files of structures rejected for various reasons.

It’s not possible to review all of the generated files in full, as this would eliminate the time-saving advantages of automated processing. However, output files of all types are spot checked for accuracy and to iteratively improve the filtering criteria. Certain output files have high potential for false positives and so we review them in full.

Formats and identifiers

Submitted files can be in one of several different formats. The most common is SDF (structure data file, a chemical structure format containing multiple structures with associated data fields). The advantage of this format is that it contains 2- or 3-dimensional structures, so we can immediately start processing the file without having to convert an identifier to a structure. This means that the final structure we deposit is more likely to exactly match the original. The disadvantage of the SDF format is that it is specialised – many users will be unfamiliar with it or won’t have software to create and display the files.

We also receive different spreadsheet formats (excel, csv, tsv) with structures encoded in text-based notation systems like SMILES or InChI. The advantage of this format is that it doesn’t require specialised software (provided the submitter has SMILES or InChIs for the compounds).The disadvantage is that the structures require conversion to SDF before processing and deposition to ChemSpider. Additionally, these formats contain information about atoms and their connectivity but lack layout information. This can introduce errors as different structure drawing packages can parse these structures slightly differently, resulting in alterations to the final deposited structure.

Filtering criteria

The criteria by which we judge chemical structures are a mixture of definitive chemical rules and less well-defined ‘rules of thumb’ based on our experience and chemical knowledge. Examples of both follow.

Empty structures, query atoms and incorrect valences

The first filter is the simplest – ChemSpider is a structure-centric database, so it’s not possible to deposit any input entries that lack a structure.

Similarly, each ChemSpider record requires a single defined chemical structure, so we exclude anything using a query atom to represent a variable atom or attachment point.

Another simple filter is to exclude structures in which atoms have invalid valences.

Charge imbalance

In general, entries in ChemSpider should represent a real-world, isolable compound. This means that we filter out structures with a non-zero overall charge. However, we make exceptions for certain examples where a counterion is generally unimportant and it’s useful to consider the charged species alone, such as choline (ChemSpider record).

Structures containing undefined stereocentres

Undefined stereocentres alone don’t represent a chemical error. However, structures like that shown below (cholesterol without any defined stereocentres) occur frequently and, although chemically valid, it’s extremely unlikely that they represent the intended structure.

Cholesterol skeleton with no defined stereochemistry

Cholesterol skeleton without stereochemistry

As a result we have a rule of thumb that excludes structures containing more than two undefined stereocentres. This is not a hard-and-fast rule, but rather an attempt to strike a balance between excluding structures like the one above and including structures where the undefined stereocentres are intentional and correct.

The count of undefined stereocentres (as determined by examining the InChI) sometimes includes cases where it is conventional to exclude stereochemical wedges. Examples include nucleic acids with no wedges on the phosphate and adamantyl groups without explicit stereochemistry – it’s unusual to draw these compounds with wedges, and users will rarely use wedges in their search. These potential false positives are filtered out and reviewed manually. A curator can then decide whether to include them in the deposition, improving the overall accuracy of the filter.

Structures containing many components

This is another rule of thumb – there’s no upper limit on how many separate components a correctly depicted chemical substance can have. However, from experience we find that excluding structures with more than four separate components removes most obviously nonsensical entries (e.g. attempts to depict alloys) while retaining the majority of correct entries.

When applying this rule, pharmaceutical molecules represent a major source of false positives because they are often multiple hydrates and/or salts with multiple counterions (e.g. Irinotecan hydrochloride trihydrate). Excluded structures that are hydrates or contain common pharmaceutical salts are flagged for human review.

Synonym filter

This filter compares the synonyms assigned to a given structure with its molecular formula and performs some ‘common sense’ checks. For example, a relatively frequent error is associating the name of a salt form (e.g., mozavaptan hydrochloride) with the structure of the free base (mozavaptan). In this case, the filter removes synonyms containing ‘hydrochloride’ because the molecular formula does not contain Cl.

SMARTS

SMARTS (Wikipedia page) is a way of describing general chemical structures. It’s based on SMILES, but has additional features allowing the specification of variable chain lengths, number of bonds, number of hydrogens, variable bond orders, or more than one potential element at a site.

We use SMARTS to identify common erroneous features in a structure. These include:

Azides and diazo groups depicted with a pentavalent nitrogen
A ‘floating’ alkane unconnected to the main structure (probably caused by an accidental click in a drawing program)
Metal carboxylates depicted as a protonated carboxylic acid with an elemental metal atom
Hexafluorophosphates (and similar species) depicted as phosphorous pentafluoride and a separate fluoride ion

SMIRKS

SMIRKS is a further extension of SMILES to depict reactions. We don’t use it to represent real reactions, but to define structural transformations – allowing us to fix simple structural errors that can be resolved by breaking and creating bonds.

One example is connecting charge-separated Grignard reagents to give a more accurate depiction:

Reconnecting Grignards

Organometallics

The difficulties of encoding organometallic structures in machine-readable formats are well documented (J. Chem. Inf. Model. 51, 12, 3149-3157). There is an ongoing IUPAC project to extend the InChI’s functionality, but for now, the challenges remain.

Every ChemSpider record is fundamentally based on an InChI, and so we are bound by the current limitations. This means that we can’t depict coordination bonds or bonds with non-integer order – any bond drawn is interpreted as a standard covalent bond with one electron contributed by each atom.

Although we generally can’t represent organometallic structures in the manner a human chemist would prefer, we still attempt to choose the ‘least wrong’ structure from various possible compromises.

Ferrocene is a classic example of this problem and illustrates several of the issues we have to consider. A few common ways to draw ferrocene are shown below (there are many more).

Common depictions of ferrocene lose bonding information when converted to mol files

Converting ferrocene structures to mol format can introduce errors in molecular formula, bond orders or valence

Most of the structures shown take advantage of extended features of chemical drawing packages in order to represent ferrocene’s bonding in a way that’s attractive and easily understandable to a human chemist. Unfortunately, once transferred to the simplified but universal mol format, some of those features are lost, resulting in nonsensical structures. Although structure D is unchanged, this representation has other problems: incorrect valence on Fe and no representation of the aromaticity of the cyclopentadienyl ligands.

We have a limited number of ways in which we can depict ferrocene and related structures in ChemSpider, none of which give an accurate representation of the bonding or a view that would satisfy an inorganic chemist. However, we can choose the ‘least bad’ of the possible compromises and allow machine readability:

Our compromise

Although this structure (ChemSpider record) doesn’t capture the hapticity of ferrocene and the charge localisation on a single carbon is inaccurate, it retains correct overall charges and valences and doesn’t show the ligands as sigma-bonded.

More generally, we apply some rules and transformations to standardise representations of organometallic structures. Many of these rules involve choosing whether to depict a metal–carbon (or metal–heteroatom) as covalent or ionic, depending on the nature of the metal and the ligand. Again, compromises are necessary when working within the limitations of machine-readable structures, but we attempt to classify ‘more ionic’ and ‘more covalent’ bonds. Some examples follow:

Disconnect oxygen from group 1 and 2 metals
Connect oxygen to all other metals
Disconnect carbon from sodium, potassium and calcium
Connect carbon to group 11 and 12 metals, p-block metals and some metalloids

As expected, general rules like these fail in certain cases. Therefore we have additional, more specific rules to cover exceptions, which we iteratively refine.

But these errors still appear in ChemSpider!

At present the filtering described only applies to new data coming into ChemSpider. The full ChemSpider database, built up over many years, certainly contains examples of every error described here. To fix these legacy errors, we intend to run the entire database through the same quality filters. This is a significant task with some specific challenges: the files requiring human review become orders of magnitude larger, the processing time and memory/CPU overhead is high, and the larger the data set the more likely we will run into false positives. In order to manage these challenges, we are taking the time to refine our processes on new depositions, and periodically checking our progress by running subsets of the full ChemSpider database through our filters. We know you need access to data you can trust, so we want to make sure we get this right. We’ll continue to update you as this project progresses, so stay tuned!

Comments Off on ChemSpider Pre-Deposition Filters

Royal Society of Chemistry Renews Partnership with ACD/Labs to Continue Providing Industry-Leading Data to Worldwide Research Community

30 Jul 2018

ACD/Labs algorithms will continue to equip ChemSpider with physicochemical property values and chemical nomenclature following ten year milestone.

Toronto, CANADA (July 26, 2018)—ACD/Labs, an informatics company that develops and commercializes solutions in support of R&D, today announced the continued collaboration with ChemSpider, a leading chemical database owned by the Royal Society of Chemistry, to continue furnishing predicted physicochemical properties and chemical nomenclature to the ever-expanding platform. For over ten years, scientists have accessed this publically-available free resource to gather information on chemical compounds in preparation of research or experimentation.

As the industry standard for physicochemical prediction software, ACD/Labs was chosen to generate property information including logP, logD (at various pHs), Lipinski rule-of-5 values, and boiling point, and to provide Name-to-structure (and vice-versa) capabilities. The renewal of the partnership further reflects the success of the platform and its continued importance as one of the most robust online chemical structure databases for the scientific community. As the platform advances, ChemSpider will continue to use ACD/Labs algorithms to provide quality insights to researchers.

“We set out with the mission of empowering researchers with a comprehensive view of chemical data to inform R&D initiatives,” said Richard Kidd, Publisher, Royal Society of Chemistry. “By working with ACD/Labs and utilizing its property information, we’ve been able to meet our users’ need for knowledge, which is reflected in our rapid growth since the Royal Society of Chemistry acquired ChemSpider ten years ago. To-date, property information populated by ACD/Labs’ algorithms has been among the most accessed on ChemSpider, and remains a key driver in our service.”

While ChemSpider has doubled the size of its database, it has remained committed to maintaining high quality data from selective sources. As the platform continues to grow, ChemSpider will use ACD/Percepta prediction algorithms and ACD/Name tools in a batch-wise fashion to populate the database and enhance publicly available chemical intelligence.

“Enabling the dissemination of chemical knowledge and providing solutions to accelerate R&D are among our top priorities at ACD/Labs,” said Gabriela Cimpan, Senior Director Sales, Europe, ACD/Labs. “ChemSpider is empowering knowledge throughout the chemical community and we feel privileged to be able to support learning worldwide.”

For more information on ACD/Percepta, visit https://www.acdlabs.com/percepta

For more information on ACD/Labs Chemical Nomenclature tools, visit https://www.acdlabs.com/name

For more information on ChemSpider, visit http://www.chemspider.com

About Advanced Chemistry Development, Inc.

ACD/Labs is a leading provider of scientific informatics technologies to R&D organizations that rely on analytical data and molecular information for decision-making, problem-solving, and product lifecycle control. Our software automates and accelerates molecular characterization, product development, and knowledge management. We integrate with existing informatics systems and undertake custom projects including enterprise-level automation.

ACD/Labs solutions are used globally in a variety of industries including pharma/biotech, chemicals, consumer goods, agrochemicals, petrochemicals, and academic/government institutions. We provide worldwide sales and support, and more than 20 years of experience and success helping organizations accelerate R&D and leverage corporate intelligence. For more information, please visit www.acdlabs.com. Follow us on Twitter @ACDLabs.

About the Royal Society of Chemistry

The Royal Society of Chemistry is the world’s leading chemistry community, advancing excellence in the chemical sciences. With over 50,000 members and a knowledge business that spans the globe, we are the UK’s professional body for chemical scientists; a not-for-profit organisation with 175 years of history and an international vision for the future. We promote, support and celebrate chemistry. We work to shape the future of the chemical sciences – for the benefit of science and humanity.

Comments Off on Royal Society of Chemistry Renews Partnership with ACD/Labs to Continue Providing Industry-Leading Data to Worldwide Research Community

Behind the Scenes at ChemSpider

26 Jun 2018

A peek at who we are, how we run the site, and how we manage data quality.

What is ChemSpider and who runs the service?

ChemSpider is one of the largest chemical databases in the world, containing data on over 65 million chemical structures. This data is freely available to the public at ChemSpider.com, a website published by the Royal Society of Chemistry.

How does the Royal Society of Chemistry support ChemSpider?

ChemSpider.com is an independent service that does not rely on direct or research grant funding. The Royal Society of Chemistry supports the website using the surplus generated by our publishing activities, allowing us to provide a sustainable and reliable service. We also generate revenue from advertising and by providing paid for web services, such as our APIs, for non-academic users. These activities help keep ChemSpider financially sustainable and help support our server costs, staff hours and development.

These services enable us to make the site available free anyone in the world, and we reached over six million unique users in 2017. These users range from school students looking for help with their homework, to researchers working in academia and industry, to general users who want to keep their chemical knowledge up to date. They come from every continent except Antarctica, and just about every country on Earth.

What goes into ChemSpider?

ChemSpider data comes from the chemical sciences community itself – submitted by researchers, databases, publishers, chemical vendors and many more.

We have two main inclusion criteria for ChemSpider data:

Machine readability – Depositors must provide structures in a machine-readable format, typically a .mol file that is interpretable by InChI – the open-source chemical structure representation algorithm.The .mol format describes how a compound is arranged, atom-by-atom and bond-by-bond. This means that it can only accurately depict small molecules with defined structures. For ChemSpider, “small” means structures up to 4000 daltons, including short peptides, oligonucleotides, and other structures. Large proteins, extended crystal lattices or long nucleotides are too big to describe sensibly in ChemSpider, but are available from other databases suited for larger molecules.
We also only accept ‘defined structures’ – compounds with exact chain lengths, fully expressed functional groups, and integer bond orders – due to the requirement to describe every heavy atom in a molecule. This means we can only accept structures for which we can generate a valid InChI.

Most ChemSpider structures are organic molecules. However, we do accept some inorganic and organometallic compounds, with specific methods for curating these.
Real compounds – We do not accept virtual or prophetic compounds.

As far as possible, we only accept compounds that have been synthesised or isolated in physical form. This means we do not accept transition states, theoretically predicted compounds, virtual compounds from vendors or prophetic compounds from patents.

Who are our data sources?

We have received data from almost 250 unique data sources, including data from chemical vendors, specialist databases, individuals, research groups and publishers. These sources cross the breadth of the chemical sciences – including biochemistry, pharmacology and toxicology, natural products, spectroscopy and crystallography. Each ChemSpider record includes links to all of the data sources for the compound, enabling users to find and to check the provenance of the data.

Our data source list is continually changing, as we find new sources of data to add and remove outdated or low-quality data sources.

We no longer accept data from other data aggregators. We have taken this step to match our quality requirements with other databases and reduce the propagation of algorithmically generated errors that can arise from prophetic sources. One example of this is Chessboardane, which originated from an optical structure recognition program interpreting a data table contained within a patent as a chemical structure. The result was an 81-carbon grid structure, erroneously identified as a complex cyclic alkane, which was deposited in a public repository and shared between multiple aggregators.

Because of this, we only seek data directly from the original sources, where we have greater certainty about the data’s provenance and accuracy, and are working to curate legacy data still within ChemSpider.

Because of examples like Chessboardane, we are cautious about accepting data from text-and-data-mined sources that depositors have programmatically extracted from text or encoded images in patents or scientific literature. After review, we have added some of the highest quality data mined sources. We will continue to review potential new data-mined sources on a case-by-case basis to ensure that their data meet our quality standards.

Automated filters

A manual check of every one the 65 million records in ChemSpider would take an individual more than 600 years to complete working round the clock – even if we only invested five minutes of curation time per record.

Instead, we run each deposition through a series of automated filters to pick out unsuitable structures, such as those with incorrect valences, unbalanced charges, or missing stereochemistry. In addition to structure filters, we also apply basic name and synonym filtering and regularly review the processed files so that we can improve our filters.

We have provided a simplified overview of this process below, and will provide a more detailed description of our filters in a separate blog post:

Structures are run through filters in KNIME. Those that fail the filters are removed and reviewed. Passed structures are deposited to ChemSpider

Curation by ChemSpider staff

ChemSpider is run by a small team of full-time curators, who work to add new compounds, remove errors, and respond to user feedback. Our staff have extensive experience of both chemical data and practical chemistry, with backgrounds in fields such as organic synthesis and art conservation, and a wealth of experience working on other Royal Society of Chemistry databases, such as The Merck Index* Online and Analytical Abstracts.

Community curation

Because we cannot review every record ourselves, we really appreciate comments or corrections from our users. The easiest way to help us improve ChemSpider is to leave feedback or email us when you spot an error. We try to act on user feedback within a few days – sooner for simpler queries. Please let us know if you find an error by leaving a comment on the relevant ChemSpider record, or by emailing us (chemspider@rsc.org).

Users wishing to get more involved can directly deposit structures and curate synonyms related to their research or work, without having to email the ChemSpider team.

We are extremely grateful for all the contributions our community curators have made over the years.

Keep using and contributing to ChemSpider

To access information on over 65 million chemical structures, go to ChemSpider.com, which is fully searchable by structure, name, or advanced query, from any device, anywhere, for free.

To deposit data, tell us about an error, become a curator, or for any other query, please do not hesitate to email us at chemspider@rsc.org

*The name THE MERCK INDEX is owned by Merck Sharp & Dohme Corp., a subsidiary of Merck & Co., Inc., Whitehouse Station, N.J., U.S.A., and is licensed to The Royal Society of Chemistry for use in the U.S.A. and Canada.

Comments Off on Behind the Scenes at ChemSpider

Introduction to the new ChemSpider website

21 May 2015

Blog post written by David Sharpe.

The ChemSpider team at the Royal Society of Chemistry is proud to announce that our new look ChemSpider website has been launched. As discussed in our last post one of the key features of this new design is to make ChemSpider work on as many devices (from desktops to mobile phones).

The ChemSpider homepage as it might appear on a desktop computer (left) and a mobile phone (right)

As the screenshots above illustrate, the difference in size, shape and the method of interacting with the page means the view of the website that you need is very different between devices. The nature of a responsive website design also means that some of the screenshots that we provide might be a little different from the view that you see when accessing the site, however the variances should be clear. We hope this results in an experience where usability and readability are not sacrificed for functionality.

What has changed? … and what has stayed the same?

To start with the things that have stayed the same: ChemSpider is still based on the same quality-data and provides mechanisms for users to supply and curate data. We also haven’t changed how the search queries work, so searches that you ran previously should still return the same results.

The key changes

1. The new page header

We’ve moved all of the old menu items into a bar at the very top of every page (1), we also display a search bar just above the main page content (2). On smaller displays you will see icons for the Quick search box, Sign In and Help items, all other options can be found under the ‘hamburger’ symbol (3).

Comparison of the ChemSpider page header on large and small screens

2. Shorter record pages

One of the biggest challenges of making ChemSpider work on a mobile is how to display all of the information that we have on a much smaller screen. I think that our solution will actually make ChemSpider better for everyone – regardless of how they view the site.

Previously, a ChemSpider record was one big long page that had basic details about the chemical structure at the top of the page, followed by a number of infoboxes that could be opened or closed and also re-ordered. This worked fine in most cases but led to situations where you had to do lots of scrolling up and down, and might not be able to spot the infobox that you were looking for. Now, we still show some information about the chemical structure at the top of the page (1) but, below that there is a single pane (2) which contains tabs (3) that allow you to select the section of the record that you wish to display. This means that it is always easy to look at some information and see the structure to which it relates.

The new page layout consists of a Compound header (1) and a Pane (2) diplaying the contents of the infotabs (3)

3. No Java, No worries

Many browsers no longer support Java applets. Good Java-free versions of chemistry tools have really started to take off in the last 18 months and the time was right to start the switch over. This means that the site now incorporates JSmol – enabling 3D structure view, CIF viewer and NMR/IR/MS spectra display as well as Ketcher and Elemental for structure input/editing.

4. Structure searches simplified

Previously, creating a structure search was a bit of a pain as you had to: open the structure editor in a pop-up, draw your structure, and then save it back into the searches pages – now our structure editors are embedded into the interface, cutting down the number of steps needed to get to your results and making it easier to tweak searches.

One particularly useful feature for anyone accessing the site on a tablet or mobile phone is the Convert Structure tab which can be used to load in a complex structure as a basis for a search, for instance using “dibenzylamine” in the structure conversion gives a structure that can be quickly elaborated to the Simpkins’ chiral base precursor amine shown in the the screenshot.

What’s next?

Hold on a moment there! We’ve only just got all of these great features into the site! I’m joking, but we will be spending time tweaking and perfecting the new design. We will then be able to focus on further development, if I were to speculate – I’d suggest that we will look at more (non-Java) tools that can be incorporated into the site to give a better experience, and new methods of improving the quality of data in our records.

In the meantime, please explore the site and do email us at chemspider-at-rsc.org to let us know what you think of the new site.

Comments Off on Introduction to the new ChemSpider website

What’s new with ChemSpider?

15 May 2015

Blog post written by David Sharpe.

Subscribers to this blog might have noticed that we’ve been a bit quiet of late. I want to assure that this doesn’t mean that we have been resting on our laurels. In fact we have been working on a whole host of improvements to ChemSpider – improving our infrastructure, developing ways to increase data quality and designing a new layout for our records.

We will discuss both the data quality work and the website redesign work in more detail in separate posts but ahead of the release of the new website design I want to provide some insight into what to expect when the changes go live.

Why are we changing the site now?

Well there are quite a few reasons:

Primarily, we need to have a site that meets the standards of the modern internet. This means that the site needs to be usable not only on a desktop computer but also on a tablet or a mobile phone. This is often referred to as responsive web design
ChemSpider has always had records that are full of lots of rich and varied types of information – which poses a challenge when it comes to presenting that information so that it is discoverable and easy to understand once found. We hope that the new layout will present data in an intuitive and clear way that will provide a better experience for everyone.
We need to move away from technologies that are not supported by the widely used browsers. Java-based tools have been an issue for users on certain platforms for a while and this is only going to get worse. For a long time we have provided non-Java structure editors alongside the Java tools (the current version of the site incorporates Elemental and Ketcher for structure drawing. This release will see the adoption of JSmol to enable 3D structure view and Spectra display widgets for devices that don’t support Java . At this time we are providing both Java and non-Java solutions but expect to phase out Java applets in the near future.
Improve the integration of ChemSpider with the wider Royal Society of Chemistry web family

Will there be any more changes to how the site works?

There will certainly be some changes to some aspects of the site due to: user feedback and bug fixes. We also what to look at how we can make more complex interfaces such as Advanced Search more usable, but we hope that there won’t be any major changes to the site.

Will all of the features that you use still be accessible?

In the main, the answer to this is; Yes! It might be that they now appear slightly differently or be accessible through a different interface. There are 2 caveats:

When accessing the site on mobile devices

The layout of a page on the smaller screens and tablets often needs to be different – wherever possible this is achieved by rearranging the elements of the page and adding new controls. But for some parts of the ChemSpider interface we realised that there wasn’t a good way to display all of the data and the only solution was not to show that part of the page on these smaller screens.
Removed features

There are a couple of features (such as the Print button) which we felt were no longer relevant in the new design or need to be redesigned to make them more usable.

When will the new site be launched?

We hope that the new site will be ready to release within the next week.

How will the changes affect you?

We hope that the transition will be smooth for everyone. Once the new design goes live you might need to refresh/clear your Browser Cache. The new design does require a modern browser with a good support of the HTML 5 specification. We will try to ensure that the site is usable on as wide a range of browsers and platforms as possible but expect that the site will not work well in older browsers such as IE7.

Will it still be possible to access the site using the old interface?

Unfortunately, the old interface will not be available alongside the new one.

How will you be able to provide feedback on the new design?

The best way to provide feedback will be to email us at chemspider-at-rsc.org

Keep an eye out for the new design – when it is made live we will write a blog post about the changes.

Comments Off on What’s new with ChemSpider?

Adding RSC CIFS to ChemSpider

09 Dec 2013

Written by Aileen Day.

We are pleased to announce that we have just imported 1047 CIFs to ChemSpider of crystal structures that were previously reported in RSC papers (and are available as ESI for those) to ChemSpider for the relevant compounds, and linked those back to the original articles and to the CCDC’s webCSD, e.g. example compound with RSC article CIF (see the CIF infobox). Since each CIF that is uploaded into ChemSpider must be associated with a ChemSpider compound, the difficult part of this task was working out a 2D molecular structure (in .mol file format) for each 3D crystal structure (in .cif file format) – which is particularly difficult because CIFs only contain information about each atomic position and not how the atoms are bonded to each other in the crystal or whether they are charged or not.
Ultimately we would like this CIF to mol conversion (and the whole upload) to be performed programmatically without human intervention. However, there is no reliable way to do that currently – although programs such as OpenBabel can be used to extract mols from each CIF, the reliability of this conversion isn’t 100%.
So as one of our student intern projects at the University of Southampton this summer (in parallel with another student intern project at Southampton University to share thesis data in ChemSpider) we used OpenBabel (version 2.3.2, run from the command line with the options -i cif inputfilename.txt -o mol -m –unique -d –AddPolarH) to extract mols for all the CIFs in the RSC archive (over 43,000 files as of June 2013) and enlisted Julija Kezina (shown below) to review the results of these conversions to ensure that only good structure and CIF pairs would be deposited to ChemSpider, and to better understand the problems in the conversion process with a view to fixing them. One problem that became immediately apparent was that because the 2D structure obtained was just a projection of the 3D structure along the a cell axis, which is not always the orientation which shows the molecule most clearly, even if they did have the write chemical connections between the atoms, so all mol structures were run through OpenEye’s cleaning algorithm before being reviewed.

Julija Kezina – Southampton University intern who examined CIF to Mol conversion

Julija compared each structure in the output mol files with those in the original CIF files to judge whether the conversion was accurate or not. In addition, as an extra check, all of the output mol structures were submitted to ChemSpider validation and standardisation platform to filter out molecules with structural problems (e.g. stereochemistry, valence or congestion issues).
Overall, approximately 30% of the CIF to mol conversions that Julija checked were good, with the right connectivity of atoms and ions (although approximately 30% of these needed the atomic positions to be repositioned to clean or tidy up the structure, either manually or using ChemDraw’s cleaning functionality). The 1047 of these mols which contain only a single molecule (without solvent molecules or cocrystals etc.) are those which have been deposited into ChemSpider with their corresponding CIFs.
The journals which had the highest successful conversion percentage were Molecular BioSystems (57%), MedChemComm (51%), Organic and Biomolecular Chemistry (44%) and Green Chemistry (44%) – the journals which in general are about small organic molecules.
Julija was working in the National Crystallography Service’s office at the University of Southampton, under the co-supervision of Professor Simon Coles, and we are grateful to them for their help and advice about the finer points of the CIF file format.

Unsuccessful CIF to mol conversions

Running and evaluating OpenBabel on such a large and varied set of structures has given us a useful opportunity to identify and categorise the most common problems encountered. Here we share these and give examples that would enable the identification of some easy fixes in the pipeline that might benefit the whole community and be used as test cases when doing so. We will report these bugs to the OpenBabel forum and because OpenBabel is open source, hope to resolve at least some of these issues in the future through collaboration with its other developers.

The following OpenBabel bugs look like they might be most straightforward to fix:

Details	Example
Category: BAD_NITRO Frequency: 233 Description: there are different ways of representing nitro groups in structure drawers – OpenBabel currently does so by producing a mol with a pentavalent nitrogen. In ChemSpider we we choose to avoid this in favour of a format with a charge-separated nitro. Solution: Allow OpenBabel to have a different output option for nitro groups to output them as shown in corrected mol file.	CIF: CCDC 194360 ChemSpider: 10001804
Category: BAD_MULT Frequency: 434 Description: Duplicate (exactly identical, including stereochemistry) molecules are present in the resulting mol file despite running OpenBabel with the –unique option (which should filter out duplicate molecules based on their inchis) Solution: Fix OpenBabel when run with the –unique option so that it works.	CIF: CCDC 229590 ChemSpider: 3915
Category: BAD_MISSINGPARTOFMOLECULE Frequency: 724 Description: Part of the molecule is missing Cause: OpenBabel doesn’t understand crystal symmetry – only the atoms in the CIF that are explicitly listed with positions are included in the resulting mol file, and those that are inferred by symmetry are not. Solution: Make OpenBabel generate the full molecule from the symmetry in the CIF file, or recommend that a script/program that can process a CIF to generate another CIF with all atoms is run before OpenBabel.	CIF: CCDC 185091 ChemSpider: 11917
Category: BAD_PARTIALOCCUPANCY Frequency: 432 Description: partial occupancy of multiple sites for a particular atom in the CIF file Cause: In CIF files sometimes positions of multiple sites are specified with occupancy less than one – OpenBabel doesn’t recognise this and assumes that the occupancy of all sites is one effectively, so that there are duplicates of some atoms or fragments in the mol file. Solution: Where the _atom_site_occupancy is less than one, group together atoms into those which are alternatives of each other (by type, proximity, and those which add up to a total occupancy of 1) and choose only one of them to include in the final mol file (that with the highest site occupancy, or if two have equal occupancies of e.g. 0.5 then pick one at random). Note that there needs to be consistency, so that if for example a C is discarded, then all of the adjoining H’s with partial occupancy are also discarded but those bonded to the C that is included are included (as in the attached example).	CIF: CCDC 854369 ChemSpider: 68005704

Many of the problems were caused by idiosynchronies or errors in the input CIFs, but these on the whole weren’t handled well by OpenBabel (e.g. by writing an error message and terminating the program) but rather, in the majority of cases went into an infinite loop and the program hung. Because of this, and because the OpenBabel conversions were part of a longer script, all OpenBabel jobs had to be run with an arbitary timeout so that if still running after this timeout they were killed, which may have discarded some valid but long-running OpenBabel jobs. We will investigate whether there is a validation program that can be automatically performed on CIFs to filter out ones with these problems (similar to the CCDC’s EnCIFer but which can be run programmatically), but it would be relatively straightforward to make OpenBabel more reliable by being able to exit nicely when it encounters these problems so that pre-validation wasn’t necessary. These problems are listed in the table below:

Details	Example
Category: CIF_NOCOORDINATES Frequency: 378 Description: cif doesn’t contain any coordinates Cause: Some CIFs contain e.g. powder diffraction refinement data and don’t contain coordinates. Solution: OpenBabel already issues an error: “CIF Error: no atom found ! (in data block:XXX)” – simply abort the program if this is found (rather than trying to continue).
Category: CIF_MISSINGLOOP Frequency: 85 Description: cif misses a “loop_” line Solution: Do an initial check that there is at least one loop_ line in the expected place before attempting to do the conversion.	CIF: CCDC 753484
Category: CIF_COMMENTEDFIELD Frequency: 36 Description: if there is a CIF field name in a commented section of the CIF, OpenBabel doesn’t ignore it and goes into an infinte loop Solution: It would be trivial to make sure that OpenBabel ignores CIF field names which are commented out (between a pair of semicolons).	CIF: CCDC 840581

The following OpenBabel bugs were the most frequent in occurence, but will be difficult to fix. They arise from the problem that the CIF format does not record charges on atoms/ions or the types of bong between them so OpenBabel needs to work them out which is hard to do correctly.

Details	Example
Category: BAD_CHARGEMISSING Frequency: 830 Description: One or more ions in the molecule have the wrong charge on them in the resulting mol file	CIF: CCDC 879075 ChemSpider: 68005707
Category: BAD_WRONGCOORDINATION Frequency: 747 Description: One or more atoms or ions in the molecule have the wrong coordination – problem observed in metal ions, S, P, Se and B	CIF: CCDC 218529 ChemSpider: 26579734
Category: BAD_BONDMISSING Frequency: 587 Description: One or more of the bonds in the molecule are of the wrong order e.g. a single bond instead of a double bond.	CIF: CCDC 926530 ChemSpider: 34226187
Category: BAD_WRONGBOND Frequency: 452 Description: Wrong sequence of single/double bonds.	CIF: CCDC 203663 ChemSpider: 238575
Category: BAD_NOCOORDL Frequency: 52 Description: no coordination to a ligand.	CIF: CCDC 218360 ChemSpider: 68005705
Category: BAD_MISSINGH Frequency: 18 Description: missing hydrogen.	CIF: CCDC 220380 ChemSpider: 21188989

There were also some problem mol files produced which either won’t be able to be fixed by OpenBabel (since they resulted from either errors or limitations of the input CIF files which cannot be fixed retrospectively) or are too difficult to fix and/or too infrequently occuring to be worth the effort:

- There were 237 cases where there were solvent molecules in the CIF (many of which have missing hydrogens, partial occupancy of the molecule or part of the molecule etc.) which give rise to spurious oxygens, fragments of molecules and radicals in the resulting mol file (see CIF: CCDC 213787 and ChemSpider record: 68005706). 148 of these cases are just water solvent molecules either with missing or detached hydrogen atoms. The poor definition of the solvent molecules is a limitation of CIF files from diffraction so it is not possible for OpenBabel to better define them in the output mol that is derived from them. However, running OpenBabel with the -r option to remove all but the largest contiguous fragment was quite successful to remove these problem solvent molecules so no further action is required to deal with this problem and this option will be used by us in the future.
- There were 81 cases where there was at least one missing hydrogen in the original CIF (or in 3 cases, all hydrogens missing) – see CCDC 259871.
- Some CIFs contain crystal structures which correspond to continuous networks rather than small molecules (e.g. polymers, MOFs, zeolites, POMs) which cannot meaningfully be captured in mol format – see CCDC 206593.
- There were a few (24) cases where the stereochemistry in the mol file obtained is incorrectly defined. However, because on the stereochemistry was well interpreted by OpenBabel and these cases were relatively few, it probably isn’t worth disturbing the apple cart to investigate these further – see CCDC 238611 and ChemSpider 9419187.

Comments Off on Adding RSC CIFS to ChemSpider

More hexagons in the plane

25 Oct 2013

Written by Colin Batchelor.

Recently I heard someone who cycled the 1400 km from John O’Groats to Lands End, with a headwind all the way, because it looked on the map as if it was downhill and hence easier. (I am grateful to Neil Swainston of the University of Manchester for this anecdote.)

You might think that “down” on the page is unlikely to be “down” in 3D space, but there is an interesting exception to this, at least for certain interpretations of “down”.

Some time ago I gave a teaser of my Sheffield talk, which is now online here and here. The mathematical meat of the talk was about redrawing sugar rings in small molecules so that they can be properly indexed by cheminformatics systems. The teaser showed a classification of hexagons so we can tell which rules to apply.

It turns out that for the hexagons we see most in practice, which are chair hexagons and Haworth hexagons, at least if the hexagon itself has its long axis roughly horizontal on the page, then if a bond points “down” on the page, when we redraw the hexagon as viewed from “above”, then the bond will still be pointing down and needs to be redrawn with a dashed bond. The same applies, mutatis mutandis, for the bonds pointing “up”.

So far, so distressingly simple. Sometimes tasks really are easier than they look. There are two more things to address, though. One is simple and involves the well-known rules for how many stereobonds you draw in any given structure (I’ve mentioned this before). The other one is tidying the molecule so that the layout algorithm doesn’t undo all your good work. This is a bit trickier and I need to look a bit more at what tools are already out there for doing this.

Comments Off on More hexagons in the plane