I’ve been reading two books this summer: Indexing: from Thesauri to the Semantic Web (Pierre de Keyser 2012); as well as Subject Access to Information: An Interdisciplinary Approach (Koraljka Golub, 2015).
 
Having worked with the Dublin Core metadata schema as a means to enhance access and retrieval of 40 years of research at the International Development Research Centre (IDRC, Ottawa, Canada), I’ve become convinced of the power of indexing as it applies to the World Wide Web. The Web Consortium (W3C) is moving towards a Semantic Web (Web 3.0), where retrieval of information becomes intelligent. Methods of indexing for context as well as content need to evolve over the next several years.
 
Both of the above books look forward to the Semantic Web, and how structures of knowledge and information will undergo change (KOS, XML and RDF- but more on this later). The books also examine user behaviour and how Google has informed search and retrieval practices and results. A Semantic Web would provide answers to search questions in a customized, or more tailored and human-centric manner than procuring, for instance, a thousand website landing pages to wade through, often ranked artificially.
 
As a member of a community of indexers and information management experts, I find along with these authors, that indexing and library science techniques have been supplanted by Information Technology experts (the IT guys in every enterprise), who are mostly oblivious to classification systems already developed and applicable to their search and retrieval problems. What I’d like to believe is that the time has come for the acknowledgement of the expertise of indexers and librarians! We exist so often in the background that we have a hard time even valuing our own experience. Re-naming ourselves as Content Management experts or Information Architects widens opportunities. Heather Hedden’s book “The Accidental Taxonomist” speaks to developments in our industry.
 
At the latest Indexing Society of Canada conference (June 2017, Montreal), a delegate from Shanghai referred to a new initiative there named “Big Data and Indexing Institute.”  We need this to take root in North America, beyond the IT guys. Indexers hold  knowledge key to interoperability of Big Data, Open Data, and Linked Open Data retrieval. Big Data and Indexing ought to be intrinsically related, (like the Ministry of Environment and Climate Change). Every business enterprise is currently engaged in re-sorting their digital assets so that decades of information can be stored (AND RETRIEVED) via the cloud. It’s called Digital Assets Management, and again, knowledge of indexing and information retrieval is key. And again, the IT guys are re-inventing the wheel because they don’t recognize the word “indexing” as a thing.
 
As you can see, just scratch the surface of this indexer, and there’s another ¾ of a (warm) iceberg beneath, just beginning to try to explain what I see happening Out There when it comes to current indexing applications.
 
I’m going to write more about the content of these books, but leave you with two blog sites devoted to DAM (Digital Assets Management), because it is a Thing that follows on the heels of cloud storage.
 
http://planetdam.org/author/90021854-ef96-4acc-8292-da83f19a3c5b
 
http://digitalassetmanagementnews.org/

 ​Indexing Pride and Prejudice: Indexing Fiction
 
This year is the 200th anniversary of Jane Austen’s death, and having read Pride and Prejudice numerous times already, I’ve decided to create an index for it. The classics are ‘classic’ for good reason. (Moby Dick may be my next indexing diversion).
 
Why index Austen’s Pride and Prejudice? Did I say it’s a fantastic read? (!!), and closer reading with an eye to an index provides the opportunity of linking nuanced first meetings with later plot developments; scenes where character is exhibited through tone of speech (Austen never tells the reader what to think), and the interlacings of relationships between family, friends and lovers.
 
Being able to refer to an index can quickly restore a new reader to the pattern of plot and relationship, where details of entails - of who is heir to what and why - are extremely important. However, present day readers may not understand what was common knowledge during the Regency period when Austen was writing. In a family of five daughters and no sons, the entrance of the distantly-related Mr. Collins is of high dramatic value. Mr. Collins makes his first entrance only on page 52 of the Bantam Classics edition (reissue 2003, based on the 1813 edition).
 
A problem for indexing is that there is no standard pdf format to work from. I’m guessing from a rudimentary Internet search, that there are thousands of editions extant in multiple languages and formats, so that whatever I create the page numbers will be irrelevant.  However, I believe that the index itself will not be irrelevant, and that publishers ought to be more interested in creating indexes for classic literature. They provide a way for new readers to orient themselves to multiple plot lines, character development, and family relationships that only become clear over time. To me, it is one future for indexing that hasn’t yet been adopted. (See button below "Index first draft")
 
The first read through of a novel is usually for storyline. If it’s a great book it can be re-read for multiple reasons such as: How is character revealed? What’s been omitted in the movie version? What was the first interaction between Darcy and Elizabeth? What was that great conversation between them at Netherfield? How do we know that Bingley truly admires Jane? Where are the first stirrings of Lydia’s narcissism underlined? At what point can we be sure that Darcy is the man who will always come through? Where do the intimations of ‘Pride’ and ‘Prejudice’ show themselves? How great is “Death Comes to Pemberley” PD James’ follow up? And then the BBC version of it? And the latest BBC Culture facebook post [https://www.facebook.com/BBCCulture/videos/1425986837485085/?hc_ref=ARQQ6l2sJi9aHEWHlzx8iuTrNbOHsgOdkaoKgAT6LB0bynYDVwYrpNAMs-uSYM1d6rI], not to mention Pride and Prejudice (and Vampires), all these keep regenerating Pride and Prejudice in ways Austen could never have imagined. An index is a no-brainer for added value.
 
More on the topic as I get past Mr. Collins entrance pages 52-73.

I was recently employed as an indexer in cataloguing a “green” library repository[1] at the International Development Research Centre in Ottawa, Canada.  My role there, along with others in the team, was to provide keywords for 8,000 research documents including records of research activities, using the Dublin Core metadata schema. Previous to this assignment I wasn’t familiar with the Dublin Core, but at the close of the project, I appreciate its scope as a vehicle for indexing and tagging collections on the web. The project was a fascinating look-in to the Dublin Core’s capacity to comprehensively index and tag the IDRC research collection, while making web searching easier and more efficient for both researchers and non-academics.

 Indexers created main keywords for repository items, first by reading documents for subject classifications and type of research, and then adding several terms describing important research results along with other types of content. I read or reviewed 3200 research papers, theses, project evaluations, reports and summaries as well as peer-reviewed journal articles and other items related to research programmes in the developing world that are funded by the IDRC. It has been an amazing opportunity to read a body of mostly hidden literature, commissioned over the past 45 years.

 As such, original documents exist in paper format. During the span of the project all have been digitized. Indexers worked from both archival hard copies as well as digitized material. The digitization and indexing of the International Development Library repository allows for forty-five years of development research results to be made available to a global public. At the same time, it provides accountability to programme donors and funders, allowing them to see some effects of their contributions.

 The Dublin Core metadata schema was customized to the needs of the International Development Library, providing bibliographical information about the research, as well as subject heading fields for key word descriptors. Keywords added to the Dublin Core records become searchable subject headings in the collection catalogue via the IDRC library database and search engine, and as well, are made available in DSpace[2] via internet webcrawlers. Checking against the OECD macrothesaurus and other controlled vocabularies for particular subject areas further refines keywords derived by the indexer.

 I learned on-the-job about controlled vocabularies. Prior to this experience it had never been required for back-of-the-book indexes, which are not web-dependent for usability. The use of controlled vocabulary is a way of particularizing and standardizing search terms so that researchers and (other) web crawlers can find the research more efficiently via the internet. At the IDRC, much of development research focuses on agriculture, climate change adaptation, global health, and poverty alleviation. Keyword vocabulary is typically drawn from the OECD Macrothesaurus, MESH (for health), CAB (for agriculture), UNESCO, UNBIS as well as others.

I discovered that problems could arise when authors are not given specific criteria for generating keywords to their own articles and reports. The situation is similar to having the author of a book create his or her own index. And while a book index is much longer, and covers more subject areas, the snapshot provided by a brief index of subject terms for a research paper can look muddled and/or quickly thrown together. Content is lost. For me, the best-case scenario happens when keywords can add up to tell a little story about what is in the paper. For example: PHILIPPINES--BOLINAO, COMMUNITY PARTICIPATION, MARINE RESOURCES, COASTAL ZONE MANAGEMENT, LIVELIHOODS, FISHING, ECOTOURISM, ENVIRONMENTAL EDUCATION, FEASIBILITY STUDIES. These keyword terms provide a snapshot for the title: Livelihood development activities in Barangay Balingasay, Bolinao, Pangasinan

 The subject catalogue for the entire collection can become distorted or inflated with author-generated keywords. For example there may be multiple terms appropriate to fish farming: fish farms, aquaculture, pond culture, aquaculture techniques, fishery management etc. But if authors suggest multiple keywords from their own vocabulary, such as fish farming, fish farms, fish farm, pond farming, fisheries, fishery management, etc, while the (CAB) macrothesaurus term is ‘pond aquaculture’, the collection catalogue can become muddled with over-generalized or inappropriate search terms. Internet web crawler efficiency is also reduced.

 Now that professional indexing of the collection ended (at least for this part of the project), current authors of research may forget to include in their keywords what seems obvious to them, such as the country of origin of the research, or terminology that has become part of their own thinking.  As terminology changes over time, these terms can become more or less familiarly known. They also become more or less searchable.   Standardized vocabulary can eliminate a great deal of confusion.

I came across this omission in more recent papers that I reviewed towards the end of the project, where author keywords were often prescribed without providing criteria or thesauri for them to refer to. As author keywords will soon become the only means of indexing the collection, over time this will create problems. If researchers commissioned by the IDRC are not familiar with pre-existing terms and categories, the searchability of the repository will eventually become diluted. In the above example of author-generated keywords, the outsider searching in the subject catalogue for the particular country or region that has been omitted in the keywords, will of course be unable to know what hasn’t been included. It’s self-evident that you can’t find what isn’t there. But it’s these invisible gaps in information that are the proper concern of professional indexing, and not the authors of research papers.

 It is understandable that author keywords are prioritized by the IDRC. After all, the authors are their own subject experts. And doing a PDF search is certainly the cheapest way to go with new material. Yet providing benchmark criteria such as the OECD macrothesaurus or other appropriate thesauri to the author, would help to keep the integrity of the subject catalogue for the collection as a whole, while retaining its efficiency for internet searchability. 

 The usefulness of a controlled vocabulary functions like the index in the back of the book. It creates a customized set of related concepts, increasing and improving searchability; in this instance, of a whole database.

  

 



[1] Kennison, R, Shreeves, SL, Harnad, S. (2013). Point & Counterpoint: The Purpose of Institutional Repositories: Green OA or Beyond?. Journal of Librarianship and Scholarly Communication 1(4):eP1105. http://dx.doi.org/10.7710/2162-3309.1105  See also https://www.youtube.com/watch?v=T2oCp6psqlE

 
[2] See ‘Google Scholar and DSpace’  http://atmire.com/website/?q=content/google-scholar-and-dspace    

Who also, as it happened, was the Newbie facilitator (sitting on my left).  So, I just want to shush any nervous Newbie qualms out there, about bunking up with congenial Others, and simultaneously thank Terry Casey for her generosity, extroverted personality, and Event Planner channelling!

After the conference, Margie Towery started a thread on the ASI-L (June 10) regarding hosting future conferences at University campuses to defray hotel costs. The hotel rooms for the Charleston conference were very roomy with sitting rooms attached, but with a door that could be closed between the sleeping area, the bathroom and the sitting room. This feature may have made sharing with a stranger in close quarters a little more graceful. One could sleep while another could stay up, or have a long shower. I discovered that the Indexers I met are caring and careful people, so the option of closing a door is I think, important. One can choose to live in close quarters, or not, and be less anxious about irritating the other person. I knew I would always be able to find a bit of quiet (even with a generous, energetic, event-planning extrovert!)

That said, now that I am no longer a complete Newbie, I could definitely entertain bunking up in a campus dorm. But as a first experience, the Indexing conference at Charleston was pretty much perfect.

With Steve Falk, Terry also facilitated and scheduled an ice-breaker activity, as well as managing the Newbies and their Buddy(ies). For conference old-timers, this is how it looked in our Newbie in-boxes:

You have been paired up as a Buddy and Newbie for the upcoming ASI Conference.  In the next week or so, please take a few moments to introduce yourselves to each other via email by answering the questions below.  That way, you will know a little something about each other when you meet in Charleston.

1.       What is your name?  (It might not be obvious from your email address.)

2.       What are your city and state and your Chapter (if you belong to one)?

3.       What types of work have you done besides indexing?

4.       What do you do for fun (besides indexing!)?

5.       What is one other fun fact about you that others would find interesting?

We have scheduled a Buddy/Newbie “get acquainted” meeting on Wednesday, April 30, at 5:15 pm at the Embassy Suites (the Conference Hotel).  We will have a short, fun ice-breaker activity, and then we will go the general Conference Welcome Reception together.  Please attend our Buddy/Newbie meeting if you can.

We look forward to meeting all of you in Charleston.

Terry Casey and Steve Falk

ASI Conference Buddy Program Coordinators

Who knew that retiring indexing persons could be so effusive and fun! The unexpected noise of gabbling circles of Indexers, introducing one to another during the ice-breaker round, was a delight to be part of.  Terry’s great positive energy paved the way for lots of strangers to have fun together. I asked her to sum up her conference experience:

The conference did, and always does, give me a boost of energy and excitement about my business.  Giving me new ideas and new connections.

That about says it all. A pleasure shared is a pleasure doubled.

In case you've not heard of this, I want to point out a couple of things so that when a client says they've decided to do their own index with PDF Index Generator for $59.95, you can right away tell them not to, and feel good about it. I’d never heard of this indexing tool until a few weeks ago, when a potential client decided just that.

After searching the net for reviews mentioning PDF Index Generator, or 'pdf generator problems' there were many references - and all but one of the articles was written by the software company itself- with no problems of course! Information about the Index-Like-Object written by professional indexers are currently hard to find. Thus, there is little competition and no countervailing arguments to accompany claims like these:

“in (4) easy steps. PDF Index Generator parses your PDF, collects the index words and their location in the PDF, then writes the generated index to a PDF or a text file you specify. The main target for PDF Index Generator is to automate the process of generating the book index instead of doing the hard work manually”

During the time I was looking at the program and thinking about writing this, I had a chance email conversation with Dwight Walker of the Web & Electronic Indexing Sig, and I asked him if he wouldn't mind taking a quick look at it too:

I downloaded the PDF Index Generator and ran it today. It is very basic. It extracts all keywords into an alphabetical list but does not edit it so the indexer would have to edit it and clean it up before pasting it back into the document or letting the app append it to the PDF. An indexer could use it as a basis then re-edit the generated index and tweak the tool.

 As you say, the index was pretty awful that PDF Index Generator created, unless the stop words were adjusted. No multi-level indexing was done just flat keyword alphabetical list, which was very poor quality.

The only good thing was the layout.

I found one review of it in PDF Index Generator 28.47-48, The Indexer by accident today. I don't have a copy of that issue of The Indexer.
- Dwight Walker

The only article which I had found, that was not written by Index Generator begins: 

Creating the initial index was a completely grueling process. It was pure grunt work, and unfortunately, there is no way around it.
PDF Index Generator will create an index list for you, but it will index every word, minus the built in ‘excluded’ words. I thought the application would be able to automate the creation of the index terms, but I quickly found out that the software can’t determine groupings of words and phrases: e.g., ‘information security management’. The Index creator can index each word individually, but it would have no way of knowing that Index Security Management is a term that you want to specifically search for, unless you manually specify it in the ‘include list’.


However, the author concludes by giving the program a rating of 7/10, which I do not believe any indexer would agree to. 
[For the full article, here is the link  http://www.curvve.com/blog/guides/2011/create-a-printable-document-index-with-pdf-index-generator/#.U6CoEI1dVgS]

Looking at the first sample index provided on the software website (see sample indexes http://www.pdfindexgenerator.com/download_samples.php) - take even a cursory glance at the A’s - and notice how names are indexed inconsistently, i.e.  Agee, Michael; Akerlof, George and then for some reason Alan Morrison. (not Morrison, Alan). This happens to names throughout the first sample index.

In the second sample index provided notice, again right off the bat, the sort of headings and locator strings created:
accountability
  personal, 2, 8, 12, 38, 55, 64, 72, 78, 85, 89, 93-96, 107, 173,    181-182, 184-185
adjourning stage of team building, 103-104
assumptions
  defined, 26, 75, 97, 109, 131, 134, 148, 151, 153, 182
  identifying, 21, 23, 33-35, 55, 59, 86, 89, 119, 134, 150, 161
  project plan template, 162

Peruse the program yourself, and you will find many ways that PDF Index Generator creates “an index-like-object” but not a usable index. But the point I'm trying to make is, the internet tourist will have to dig deep to find reasons not to buy PDF Index Generator. To the unsuspecting debt-ridden first-time author it might look like a pretty good object! Even though the flaws might be obvious to indexers, to the general public the faults are NOT obvious. If a person doesn't know how a proper index ought to function, the Index-Like-Object can look just like a usable index.


That’s why I think it is important to have a couple of things to say to the wary author who thinks that for $59.95, they are getting a deal (while pdf generator steals your fee). It’s all the same to them, because, when they search the net looking for answers, there’s no evidence to the contrary.

Scroll down below, for the Seth Maislin article which Dwight located (as pdf ) and another by Jon Jeremy at this link http://www.anzsi.org/UserFiles/file/Feb%2010.pdf (page 8 ANZSI Newsletter)

For the ASI conference, David Ream presented an update of what the ASI Digital Trends Task Force unit have been up to for the past year (or three)- completing a coding language of the structural components of an index that software developers can adapt for EPUB platforms. At least, that is how I understood his presentation. Early on in it, Mr. Ream threw up this image (above) in the powerpoint (for about 3 seconds) and when I saw it, it kind of blew my mind a bit. I asked him to go back to it during his talk, and I want to show it again here, in a screen capture. It's a chart of results drawn from this Levtech resource http://www.levtechinc.com/pdf/Using%20BNA%20Indexes%20study.pdf. It illustrates time taken to access information through use of an index or keyword search. The INDEX clearly wins! 

As a researcher, when I have come upon a beautiful index, it elicits pure emotion. I know this sounds weird to anyone who is not a researcher or an indexer - but pure delight and admiration. Wow. A good index is SO useful!! The beautiful ones are distinct I think, and somewhat rare. But any functional index is waaay better than none! The chart shows clearly how quickly an index can make information accessible, and I was actually a bit thrilled to see it, because it becomes exasperating trying to explain to people the difference between a keyword search and an index (we've all been there too many times). It illustrates the time taken doing it both ways, as you might say. Maybe I will whip it out the next time someone asks me!  Feel free to add it to your websites! 

Another thing that turned me on (as an Indexing person) was a 'User case scenario 2' that the Task Force had posited for how to locate the web index or EPUB index on screen. I loved the idea of having it pop-up as you hover over a locator, to indicate other related entries. Does that make sense to anyone else? I see it as a kind of transparent overlay- you can see through it, to what you are reading in the ebook. It appears and disappears as needed. It could also pop up as a sidebar, or as an editing bubble like on Word. 

Anyway, my preferences aside, the work they have done is amazing, and translates into something really useful. I am comparing it to the very little I know through watching a webinar presented by the ASI last October: Converting Legacy Print Book Indexes for Use in ebooks. It seems to me that the DTTF folks have gone way beyond what was being discussed and wrangled through only 6 months ago.  Someone said it was like the era when no one knew whether the common language of video would be VHS or Beta. It was a kind of relief to me to see how far they had come in working out a code for coders. These are the thoughts of an indexing naif. I'd be happy to learn others' perspectives.

Indexing Best Practices was the seminar I attended the second day of the ASI conference. I want to mention this because it gave me insight into the kind of committed committee work that ASI members do in order to maintain the high functioning of the ASI.

This group met many times over the past year, with much energy expended in co-ordinating meeting times and places, not to mention, recipes of food eaten at meetings! (Did I mention that Indexers are Foodies?) The committee included Linnea Dwyer, S. Anne Fifer, Pat Rimmer, Donna Shear, and Margie Towery. NISO technical report and guidelines were referred to as some standards of the industry, as well as the Chicago Manual of Style 16. We were reminded that 'standards' do not mean that everyone does it.


The group has written the brief of Best Practices as a 'living document'- it is not finalized, and welcomes contributions from Indexing Professionals over a 90 day commentary period. What is brilliant about this document is that it attempts to loop Appendices on specialized subjects back in to the main document. I'm including it here as a pdf file because of its value as a way to organize my own thinking, and before it changes with new contributions from the pros. Eventually, the various specialties will be able to be downloaded separately from the main document.


There were many experts in the field who participated in the talk back, question and answer period of the workshop. One of the illuminating comments from the indexers of children's educational publications revolved around terminology used in national "common core" educational units that had to show up as keywords as part of state standards. The discussion focussed on how these words can become indexing headings and subheadings, usable for children, who may not easily access or understand the index. 


The committee had begun to assemble this information in the Appendix A- Children's Books and K-12 Textbooks. I include this excerpt to show just a very small portion of the work of this group and how useful their assembled information already is, to someone like me who is new to the trade:


Main Headings
a. Main headings in elementary level books may be adjectives if they are vocabulary
words, and if the publisher or author requests that all vocabulary words be indexed.
b. Typical children's vocabulary must be used as main headings and should be cross-referenced
to unusual terms used by the author. For example, cars should be used rather
than vehicles. Vehicles should be cross-referenced to cars.
In addition, textbook indexes must reflect the wording in Common Core State Standards
in addition to actual wording in the text if it is not the same. Cross-references should
direct the user between the two. For example, in a Social Studies book the development
of towns is discussed. CCSS wording is "settlements." Both "towns" and "settlements"
should be main headings with a cross-reference from settlements directing the user to
towns.
Main headings are commonly boldface type and have initial capital letter, but
formatting must follow the style guide of the publisher.


This committee inspires confidence in ASI standards of practice as a 'living documents' that are liveable.

I've finally got a minute to pore over* what happened during the four days of the ASI conference. I want to remember a bunch of things that happened and share a couple of them before too much more time passes.
First of all, as a newcomer to indexing, I thought that the 'Newbie' full day classes given by Enid Zafran and Fred Leise were very thoughtfully presented. There were 16 of us, and it was so great to meet others who were starting out.
In my city of Ottawa, Canada, there is no Indexing Chapter. However, I am extremely fortunate to have the amazing Clive Pyne of cpynebookindexing as my mentor. 

I've discovered from spending four days with Indexers what an eclectic bunch of talented people they are! I count myself lucky to have spent time with so many smart, empathic, quick-witted, intense and thoughtful people.

They are a bit like artists (which I will get into in another post) in that they spend a lot of time working in isolation, then they come out of it and talk like mad about what they've been up to. Then everyone gets emotionally overwhelmed at about the same time, and then we all go to bed by 11PM. There are a lot of Indexers who are also Foodies, so it was great to find out that food would be ample and tasty with regular breaks for tea, coffee, green juice and cocktails at 5.

The main point of this post is to remind other indexers that Enid Zafran's Advice from A-Zafran is going to be turned into a book, so if you want to read it online or print it out (I have just done this) now is the time! There is so much excellent background info, and need I say that Enid is a wonderfully easy-going writer and presenter.

*  
Pore over vs. pour overThe phrase meaning to study carefully is pore over. It comes from a little-used sense of the verb pore—namely, to meditate deeply. In modern writing, this sense of pore rarely appears outside this phrase.

Pour over is of course a meaningful phrase in its own right, but it has nothing to do with studying. It’s what you do, for example, with milk to a bowl of cereal.

The American Society for Indexing conference happens at the end of this month in Charleston South Carolina. As a new indexer I look forward to meeting some great indexing professionals.