DtSearch man page on HP-UX

Man page or keyword search:  
man Server   10987 pages
apropos Keyword Search (all sections)
Output format
HP-UX logo
[printable version]

DtSearch(special file)					DtSearch(special file)

       DtSearch — Introduces the DtSearch text search and retrieval system

       DtSearch	 is  a	general	 purpose text search and retrieval system that
       serves as the text search engine for the DtInfo browser in  the	Common
       Desktop Environment (CDE). DtSearch utilizes a full text inverted index
       of natural language words and stems. Both queries  and  documents  have
       been  internationalized	for CDE single- and multi-byte languages, with
       provision for the definition of custom languages.  Queries  are	simple
       text  strings  that  can optionally include full boolean specifications
       with a simple intuitive syntax. Results of searches can be ranked  sta‐
       tistically.  Document retrievals can include information for highlight‐
       ing query words in retrieved documents.

       DtSearch consists of two major functional areas.	 The first is a set of
       offline build tools that:

	  ·  Create searchable databases.

	  ·  Index  the user's text files and load the resultant search infor‐
	     mation into the databases.

	  ·  Maintain the databases.

       The second functional area is an online search API. It provides a  sim‐
       ple  interface  to  the search engine to facilitate user-written search
       and retrieval programs. The API consists of a set of functions compiled
       into  the library libDtSearch, with function prototypes, constant defi‐
       nitions, and data structures defined in Search.h. DtSearch  includes  a
       sample browser source program, dtsrtest.c, to demonstrate API usage.

       Information  and	 error	messages  in  both functional areas, including
       those appended to the online API MessageList, are generated from a sin‐
       gle DtSearch Message catalog, dtsearch.cat. The source for this catalog
       is dtsearch.msg.

       Each DtSearch database is associated with a single full	text  inverted
       index.  In addition, each database can be partitioned into logical sub‐
       sets of documents called "keytypes" by a naming convention of the data‐
       base  keys. The search engine can open multiple databases and users can
       specify any combination of databases and keytypes for each query,  thus
       providing  a  two  tier	search	capability.  Users can further qualify
       searches by restricting the search return list by date ranges and maxi‐
       mum number of documents returned.

       DtSearch	 is  written  in  ANSI	Standard  and  POSIX  compliant C. The
       DtSearch online search API is not  reentrant  (not  "thread-safe")  and
       must therefore be directly linked into the user-written search program.
       The DtSearch API will increase the size of  a  browser  search  program
       from 100K to 200K bytes depending on which functions are called.

   Database Names
       Databases  consist  of  a set of binary and ASCII files whose names are
       the 1- to 8-character ASCII database name specified to  the  dtsrcreate
       command,	 a period (.), and a 1- to 3-character ASCII file name suffix.
       Executing dtsrcreate will create and  initialize	 these	files.	 After
       creation, databases are always identified by the 1- to 8-character name
       string used in dtsrcreate. The database names dtsearch and austext  are
       reserved and may not be specified.

   DtSearch Languages
       Each database is associated with a single natural language. Unlike con‐
       ventional locales, a DtSearch language includes code  set  presumptions
       and,  most  importantly, linguistic parsing and stemming rules to iden‐
       tify indexable terms in a text stream. A DtSearch language is specified
       when a database is created. Developers can also define custom languages
       with special code sets and linguistic rules. See "Language Parsing  and
       Stemming" in this man page below for details.

   Database Types
       The  API	 can be used simply as a search engine, referring to documents
       only through the inverted indexes. Alternatively,  a  database  can  be
       configured  to  store  actual  document	text in compressed format in a
       repository efficiently accessible  to  the  engine.  The	 configuration
       options	that  indicate	these alternatives are referred to as database
       types and are specified to dtsrcreate at database creation time.

       A field called the "abstract" is included in the fzk file for each doc‐
       ument  loaded  into a database, and is included on the Results list for
       each document in a successful search. When documents are not stored  in
       a  repository,  the  abstract  typically specifies a file name, URL, or
       other reference useful to the browser.  It  can	also  include  summary
       information  viewable  by  users	 to  help  them	 select	 documents for
       retrieval and display.

   Offline Build Tools
       dtsrcreate creates and initializes new databases or reinitializes  pre‐
       existing databases. Textual data is loaded into databases by the execu‐
       tion of two programs. dtsrload creates a	 database  object  record  for
       each  text document, and dtsrindex creates the full text inverted index
       of words and stems for each object record.  Based  on  unique  database
       keys  for  each object, dtsrload and dtsrindex can also serve as update
       programs for preexisting databases.

       The input to the load and index programs is a canonical text file  with
       a .fzk file name suffix. The format of fzk files is sufficiently simple
       that they can be generated manually. In addition, DtSearch  includes  a
       utility	program, dtsrhan, which can generate a correctly formatted fzk
       file for some kinds of text documents.

       Several other utilites provided in the distribution package  are	 suit‐
       able  for  extracting summary database information, including dtsrdbrec
       and dtsrkdump.

   Argument Conventions
       Optional command line arguments are specified with a dash (-) and typi‐
       cally  a	 single character argument identifier. Some required arguments
       also use the dash convention. Unless specifically indicated  otherwise,
       dash  arguments	may  be	 specified in any order. Where values are used
       with dash arguments, they must be directly  appended  to	 the  argument
       without white space.

       Optional	 arguments precede required arguments. Non-dash required argu‐
       ments must usually be specified in the order  indicated	by  the	 usage

   Parsing and Stemming
       Word  parsing  is fundamental to DtSearch operations at both index time
       and query time. Linguistic  parsing  algorithms	filter	incoming  text
       strings	into  sequences	 of  word  tokens  for	each natural language.
       Depending on the language, word tokens may also be processed into  stem
       tokens.	At index time each linguistic token, or term, in a document is
       stored in the inverted index. At search time  queries  are  parsed  for
       linguistic terms and used to access the documents that contain them.

       Each  database  is  assigned  its own DtSearch language identified by a
       language number at database creation time. A language number determines
       the  parsing  and  stemming  algorithms to be applied to the database's
       text and queries. Internal DtSearch algorithms are  supplied  for  sup‐
       ported  languages including several European languages and Japanese. In
       addition a user exit mechanism permits developers to provide their  own
       custom language algorithms for a database.

   Language Files
       Language	 algorithms  often  use	 various  word lists. Typically, these
       lists are stored in language files for easy maintenance, with the  type
       of  list	 identified  by	 the  file  name extension. Language files are
       opened and read into internal tables when the offline programs initial‐
       ize  or	when the DtSearchInit online API function is called. Some lan‐
       guage files are required and initialization will return fatal errors if
       they are missing. Some language files are optional and associated algo‐
       rithms will be silently bypassed if they are missing.  Files  for  sup‐
       ported  languages  may  be edited to provide database specific enhance‐
       ments. At open time, database specific files supercede generic language

   General European Parsing Rules
       The currently supported European languages are

       0       English, ASCII character set
       1       English, ISO Latin-1 character set
       2       Spanish, ISO Latin-1 character set
       3       French, ISO Latin-1 character set
       4       Italian, ISO Latin-1 character set
       5       German, ISO Latin-1 character set

       If  not	otherwise  specified, dtscreate will initialize databases with
       language number 0. Note that all supported  European  languages	use  a
       single-byte  encoding  method, with the ASCII code set as a proper sub‐

       Parsed text, including both queries and indexed text in	documents,  is
       case insensitive in supported European languages.

       In supported European languages parsing is accomplished with the Teskey
       algorithm, which partitions a character set into	 characters  that  are
       always parts of words (concordable), characters that are never parts of
       words (nonconcordable), and characters  that  may  be  parts  of	 words
       depending  on context (optionally concordable). Typically, alphanumeric
       characters are concordable. Whitespace and most punctuation is  noncon‐
       cordable.  Slashes are examples of characters that may or may not sepa‐
       rate words depending on context. The essence of the  parsing  algorithm
       is  "optionally concordable characters preceding concordable characters
       are concordable; otherwise, they are nonconcordable". For example, UNIX
       directory names of the form /usr/local/bin would be considered just one
       word, but slashes in isolation would be discarded as nonconcordable.

       The parsing algorithm does a table lookup to determine the concordabil‐
       ity  of	characters.  The  tables are arrays of the characters for each
       code page supported by the algorithm. Currently	7-bit  ASCII  and  ISO
       Latin-1 are supported.

   Words Not Indexed
       Several	additional parsing rules are applied to prevent indexing mean‐
       ingless terms. These  terms  include  common  prepositions,  indefinite
       articles,  and  nonlinguistic  text  strings  such  as formatting tags,
       sequences of hexadecimal dump characters, list identifiers, etc.

       Tokens whose lengths are less than a minimum word size or greater  than
       a maximum word size are discarded. The default minimum and maximum word
       sizes can be overridden in dtsrcreate.

       Similarly words found in the "stop list" file for the database are dis‐
       carded.	Stop  lists  are  external, editable language files. Each sup‐
       ported European language is provided with a default stop list.

       Words found in an "include list" file are forcibly indexed even if they
       would otherwise be discarded. Include list database files are optional;
       no defaults are provided.

       When specified for a language, individual parsed words  will  be	 "con‐
       flated"	or  mapped  into their "stem" form, a new word that represents
       the etymological root of the original word.  A  default	null  stemming
       algorithm  is used for languages that are not otherwise provided with a
       supported stemmer. The null stemmer returns the original	 word  as  its
       own  stem.  Both	 words and stems are stored in the inverted index. API
       searches can be specified for either words or stems, but the two search
       methods	are distinguished only when real stems have been stored in the
       inverted index.

       In the  supported  European  languages  stemming	 can  be  accomplished
       heuristically  or  by dictionary lookup. The heuristic algorithms typi‐
       cally remove affixes in a language-dependent way. Affix lists are  usu‐
       ally stored in language files. Currently stemming is supported for Eng‐
       lish languages 0 and 1, Spanish language 2, French language 3,  Italian
       language 4, and German language 5.

       Two  Japanese  DtSearch languages (numbers 6 and 7) are supported. Both
       use the same packed, Extended UNIX Code (EUC) character	set.  The  two
       languages  differ  only	in  the technique used to parse compound kanji
       words. All validly encoded text for supported Japanese languages incor‐
       porates	ASCII  encoding as a proper, single-byte subset. The supported
       Japanese languages use the null stemmer.

   Kanji Compounds
       Individual kanji characters are parsed as single	 words.	 In  addition,
       for  language  number 6 all possible kanji substrings (pairs, triplets,
       etc.)  found in any contiguous string of kanjis will be parsed as  com‐
       pound kanji words, up to a maximum word size of 6 kanji characters. For
       language number 7, only kanji substrings listed in the jpn.knj language
       file  may be treated as compound kanji words. At offline index time all
       possible individual kanjis and  kanji  compounds	 for  a	 language  are
       stored in the inverted index. At online search time kanji substrings in
       the query are treated as single query terms and are not compounded fur‐

   Japanese Code Sets
       The supported packed EUC character set consists of four separate multi‐
       byte Code Sets. Code Set 0 can be either	 7-bit	ASCII  or  7-bit  JIS-
       Roman.  The  first  and	only byte of a character in Code Set 0 is less
       than 0x80. Substrings of Code Set 0  in	supported  Japanese  text  are
       parsed	into  individual  words	 with  the  European  language	parser
       described above. Minimum	 and  maximum  word  sizes,  stop  lists,  and
       include	lists will be used as in European languages if provided with a
       Japanese database.

       Code Set 1 is JIS X 0208-1990. The two-byte characters in  Code	Set  1
       always  begin with a byte greater than 0xA0 and less than 0xFF. Symbols
       and line drawing elements are not indexed. Hirigana  strings  are  dis‐
       carded  as  equivalent  to  stop	 list  words. Contiguous substrings of
       katakana, Roman, Greek, or cyrillic are parsed as single words.	 Indi‐
       vidual  kanji  characters  are  treated as single words with additional
       kanji compounding depending on language	number,	 as  described	above.
       Characters  from	 unassigned  kuten  rows  are  treated as user-defined

       Code Set 2 is halfwidth katakana. The two-byte characters in Code Set 2
       always  begin  with the unique byte 0x8E. Contiguous strings are parsed
       as single words.

       Code Set 3 is JIS X 0212-1990. The three-byte characters in Code Set  3
       always  begin with the unique byte 0x8F. Parsing is similar to Code Set
       1: discard symbols, etc., contiguous strings of related foreign charac‐
       ters equal words, and individual kanjis and unassigned characters equal
       single words, with additional kanji compounding depending on  language.
       Kuten  row  5  is  treated  as  katakana; undefined rows are treated as

   Custom Languages
       All language dependent data structures and functions are referenced  by
       fields  in  the	main internal DtSearch structure for databases (DBLK).
       The same structure is used for offline build programs as well as online
       API  search  functions.	Language processing is initialized database by
       database by an internal language loader function which stores values in
       DBLK  fields. A database whose language number is not supported is pre‐
       sumed to be associated with a  custom  language.	 A  special  function,
       load_custom_language,  is called to initialize language fields for cus‐
       tom languages. The default load_custom_language merely returns an error
       code.   However,	 developers can link in their own load_custom_language
       function, which will be called to initialize the DBLK fields needed  to
       parse  and  stem	 one or more custom languages. Values required for the
       language fields of a DBLK are specified in DtSrAPI(3).

       dtsrcreate(1),  dtsrdbrec(1),  dtsrhan(1),  dtsrindex(1),  dtsrload(1),
       dtsrkdump(1), huffcode(1), DtSrAPI(3), dtsrfzkfiles(4), dtsrocffile(4),
       dtsrhanfile(4), dtsrlangfiles(4), dtsrdbfiles(4)

							DtSearch(special file)

List of man pages available for HP-UX

Copyright (c) for man pages and the logo by the respective OS vendor.

For those who want to learn more, the polarhome community provides shell access and support.

[legal] [privacy] [GNU] [policy] [cookies] [netiquette] [sponsors] [FAQ]
Polarhome, production since 1999.
Member of Polarhome portal.
Based on Fawad Halim's script.
Vote for polarhome
Free Shell Accounts :: the biggest list on the net