You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opennlp.apache.org by bg...@apache.org on 2016/11/16 09:10:57 UTC

[01/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Repository: opennlp-sandbox
Updated Branches:
  refs/heads/master 9aa270c11 -> 1f97041ba


http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/189un_CEDAW_C_2002_II_3_ADD2.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/189un_CEDAW_C_2002_II_3_ADD2.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/189un_CEDAW_C_2002_II_3_ADD2.txt
new file mode 100644
index 0000000..994d464
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/189un_CEDAW_C_2002_II_3_ADD2.txt
@@ -0,0 +1,2 @@
+
+Committee on the Elimination of Discrimination against Women Twenty-seventh session 3-21 June 2002 Item 5 of the provisional agenda* Implementation of article 21 of the Convention on the Elimination of All Forms of Discrimination against Women Report provided by specialized agencies of the United Nations on the implementation of the Convention in areas falling within the scope of their activities ** The document was submitted late to the conference services without the explanation required under paragraph 8 of General Assembly resolution 53/208 B , by which the Assembly decided that , if a report is submitted late , the reason should be included in a footnote to the document . Note by the Secretary-General** Addendum World Health Organization On behalf of the Committee on the Elimination of Discrimination against Women , on 17 April 2002 , the Secretariat invited the World Health Organization ( WHO ) to submit to the Committee a report on information provided by States to WHO on the
  implementation of the Convention on the Elimination of All Forms of Discrimination against Women , in areas falling within the scope of its activities , which would supplement the information contained in the reports of the States parties to the Convention to be considered at the twenty-seventh session . Other information sought by the Committee refers to activities , programmes and policy decisions undertaken by WHO to promote the implementation of the Convention . The report annexed hereto has been submitted in compliance with the Committee 's request . Annex Report of the World Health Organization to the Convention on the Elimination of All Forms of Discrimination against Women at its twenty-seventh session Comments on the health of women in States parties presenting initial reports : the Congo , Saint Kitts and Nevis , and Suriname Introduction Women 's health is an issue given some prominence both directly and indirectly in the Convention on the Elimination of All Forms of Dis
 crimination against Women and the potential negative effects of discrimination on the health of women is a theme that appears in various places . Article 12 explicitly sets forth the obligation of States Parties to take all appropriate measures to eliminate discrimination against women in the field of health care in order to ensure , on a basis of equality of men and women , access to health-care services , including those related to family planning . Other articles strengthen and broaden this article , calling for the elimination of discrimination against women in a number of health-related fields , for example , articles 10 ( education ) , 14 ( rural women ) and 16 ( marriage and family relations ) . The General Recommendations , likewise , have expanded and specified other issues which the Committee on the Elimination of Discrimination against Women ( CEDAW ) also believes it is important to monitor for discrimination and which are related to health , for example , Nos. 12 ( 1989
  ) and 19 ( 1992 ) on violence against women , No. 14 ( 1990 ) on female circumcision , No. 15 ( 1990 ) on avoidance of discrimination against women in national strategies for the prevention and control of AIDS , No. 18 ( 1991 ) on disabled women and No. 24 ( 1999 ) , the Committee 's most extensive comments on women 's health . General Recommendation No. 24 , completely related to health , is of particular importance . This recommendation places the health of women in a comprehensive framework and takes a lifespan approach , recognizing the importance of the age-specific needs of women at various points throughout life . The World Health Organization ( WHO ) is pleased to submit this report with information which may complement the reports of States parties and draw attention to some issues related to women 's health that may merit the special attention of the Committee . Information for each country is grouped under various headings \u2014 basic data , children , adolescents , women 
 , older women and conclusions \u2014 selecting , in each case , information particularly relevant to the health of women of all ages . It is to be regretted that data on a number of issues important to the health of women , particularly poor women , tends to be inadequate or absent altogether from much of the standard published information about women in many countries . For example , \u2022 While WHO and a number of other United Nations agencies are deeply concerned about the issues of trafficking in women and children and there are suggestions it may be an important issue to some of the countries reporting to CEDAW , there is little systematic country-specific information available about either the volume or the threats to the health and well-being of women who are victims of trafficking ; \u2022 Two other important issues which are underrepresented are the overall incidence , impact and male-female distribution of child labour and occupation-related health problems of women in both formal
  and informal work settings ; \u2022 Data is scarce on women with disabilities and other women who live or work under circumstances that place them at particularly high risk of disease or disability , such as , inter alia , migrant workers , refugees and other socially displaced women , and sex workers . These are issues which could usefully be raised in connection with every country . Finally , it would be extremely useful if the Committee should wish to emphasize the importance of women 's health beyond the long-standing concern with reproductive health . For example , many countries have little systematic disaggregated data available on women 's health issues such as : \u2022 Communicable diseases , including malaria and tuberculosis ; \u2022 Cardiovascular diseases ( increasingly important as killers of women ) ; \u2022 Mental health problems ; \u2022 Lifestyle-related disability and death including substance abuse ( smoking and alcohol ) ; \u2022 Osteo-arthritis , auto-immune diseases and others
  which are particularly important as contributors to disability and death of older women . WHO , as a member of the United Nations system , takes the CEDAW reporting process seriously and considers it to be an important opportunity to strengthen work related to women 's health . With a view to strengthening this work , the Department of Gender and Women 's Health of WHO is preparing a publication intended to help staff working with counterpart Governments in the preparation of the health aspects of the country report to the Committee . The document has been developed with input from all WHO regional offices and technical departments and in collaboration with Rights and Humanity . It is hoped the members of CEDAW will also provide input to the draft during the twenty-seventh session . The Congo General The Congo ( Brazzaville ) has a total population of 3 million people1 and a population growth rate of 3.1 per cent.2 The population is almost equally divided between women and men , wi
 th the women making up 51 per cent of the total . Life expectancy at birth is comparatively low . However , as in many other countries , women have longer life expectancy ( 53 years ) than men ( only 50 years).3 The majority of the people of the Congo live in urban areas ( an estimated 65.4 per cent in 2000).4 The urban population growth rate for 2000-2005 was estimated to be 3.96 per cent per year , slightly lower than the regional ( Middle Africa ) figure of 4.40 per cent per year for the same period.5 According to World Bank figures , the Congo falls in the low income category.6 In 1999 , the gross national product per capita was US$ 670.7 Total expenditure on health in 1998 was estimated to be 3 per cent of the gross domestic product , slightly higher than in 1997.8 The overall situation of the Congo is challenging in terms of health and all aspects of human development . This fact was commented upon in some detail in the concluding observations of the Committee on Economic , So
 cial and Cultural Rights.9 Civil war led to massive population displacement as well as the disruption of the national economy and the production and marketing of food . This had a profound , negative impact on the situation of daily life . Health and health systems , likewise , have been affected . Children Nearly half of the population of the Congo ( 46.3 per cent ) is under 15 years of age.10 Almost one quarter ( 23 per cent ) are adolescents between the ages of 10 and 19.11 The infant mortality rate ( probability of an infant dying before 1 year of age ) is 87.2 per 1,000 live births for girls and 100 for boys.12 The child mortality rate ( probability of a child dying before reaching his or her fifth birthday ) is estimated to be 122 for girls and 134 for boys.13 For children who survive , problems of health and nutrition continue and are serious . Malnutrition runs high , immunization coverage is low and the impact of AIDS is increasing . The proportion of children below 5 years
  of age suffering from wasting is about the same as the regional average of 10 per cent.14 However , 45 per cent suffer from moderate and severe stunting , 8 points higher than the regional levels and 13 points higher than the world levels.15 Immunization levels reach only 50 per cent or lower . Unfortunately , neither nutrition nor immunization figures offer data disaggregated by sex . Table 1 Child immunization rates in the Congo , 1995 and 200016 HIV/AIDS is having a devastating impact both directly and indirectly on the children of the Congo . At the end of 1999 , 4,000 children below 15 years of age were living with HIV and 35,103 had been orphaned by the epidemic , losing either their mother or both parents to AIDS.17 One must assume that the numbers have increased significantly since then . Comment These data have important implications for national policy as well as for the design and development of public services , particularly in fields such as education and health . With
  little disaggregated data available it is difficult to know whether there are problems of particular seriousness for girls as contrasted with boys . The Committee may therefore wish to emphasize to the representatives of the Congo the importance of ensuring equity and equality in the promotion of child health and the necessary infrastructure . The impact of stunting , now affecting 45 per cent of the children under 5 , is lifelong and can be irreversible . Similarly , the low immunization levels mean that the majority of Congolese children are exposed to avoidable and , in some cases , fatal disease and disability . The cost of these low immunization rates is high for both the individual and the nation in premature death and loss of dignity and productivity through lifelong disability . In view of this fact , any discussion of health should emphasize the urgency of appropriate management of health and nutrition in the early years of a child 's life , both male and female . Adolesce
 nts In the field of health , early pregnancy and HIV/AIDS are the two most conspicuous issues about which there is solid data indicating the need for attention and action . During the period 1995-2000 , there were 141 births per 1,000 females from 15 to19 years of age,18 about the same as the regional figures ( 143 per 1,000)19 but considerably higher than desirable in terms of the health and development of both the mothers and the children . Figures are not available on adolescents with HIV , but it is noteworthy that in the 15 to 24 age group , an estimated 6.5 per cent of girls/women are HIV positive , while only 3.2 per cent of boys/men are so.20 The data on teen pregnancy , HIV infection and disparity in rates of infection between boys and girls all demonstrate the urgency of action to improve the availability of and access to appropriate health information and services , particularly addressing the reproductive health needs of adolescent girls . Bearing in mind that the gross 
 secondary school enrolment ratio of girls is only 45 per cent,21 such action would need to be focused on mapping the activity of adolescent girls to determine where and how best to reach them and help them to be empowered to take appropriate measures . The issue of the impact of civil war and its effect on many aspects of life has been highlighted in discussions of the Committee on Social , Economic and Cultural Rights . Related to this , an additional issue relevant to the health of adolescent girls in the Congo is the question of their involvement in past or present military activity as combatants or part of the support system for combatants . Unfortunately , there is little relevant hard data available . Comment Threats to the health of adolescent girls are many , but among the most serious are those related to sexual and reproductive health and behaviour . Data related to teen pregnancy and HIV infection suggests that gender-prescribed roles place girls at disproportionate risk 
 of pregnancy and ill health and leave them little latitude to protect themselves . In this connection and in line with paragraph 18 of General Recommendation No. 24,22 the Committee may wish to pursue discussion with the representatives of the Congo , focusing on availability of gender and age-specific reproductive health information and services . Activity to promote the evolution and acceptance of more equitable and balanced relations between the sexes could contribute to the effectiveness of such programmes . In line with the stipulations of the Convention on the Rights of the Child and the healthy development of girls , all involvement of girls in military activity should be firmly discouraged . The Committee may therefore wish to explore the availability of programmes of rehabilitation and efforts to prevent the involvement of girls in military or quasi-military activity which may be damaging to their mental or physical health and development . Women Women share with the whole 
 community the burdens resulting from life under conditions of poverty , insecurity , civil war and social disruption . Women are , in fact , disproportionately affected by all these conditions , which place their mental and physical health at great risk . Reports of both the Human Rights Commission and the Committee on Economic , Social and Cultural Rights have called attention to widespread problems of domestic violence , women 's victimization by armed men , serious disruptions of food production and the paucity of health services in both urban and rural areas . All of these must be assumed to contribute to the generally high fertility rates ( 6.3 for the Congo in 200023 as compared with a global level of 2.68)24 and low levels of health among women . However , specific data is seriously limited . Added to this already difficult situation , HIV/AIDS is moving rapidly through the population . At the end of 1999 , it was estimated that 6.43 per cent of the population was infected , 
 45,000 women and 37,000 men.25 Women at risk of special health problems are those among the population of concern ( refugees , asylum-seekers and returned refugees ) . As of the end of 1999 , they totalled more than 63,000 ( 53.8 per cent of the total population of concern).26 Comment Serious attention is needed to find ways progressively to address the special health risks , needs and vulnerabilities of women in the Congo , both in the general population and in refugee and other temporary living situations . Wherever they are located , the women are entitled to reasonable access to basic health information and services and adequate protection to ensure preservation of their dignity and their physical integrity . The Committee may , therefore wish to enquire about both immediate and longer-term plans to carry out programmes and build community-based infrastructure and the necessary support systems to ensure that the women of the Congo have access to increasingly effective health ser
 vices in accordance with standards set forth in the General Recommendation on health . Given the extreme hardships that much of the population has undergone in recent years , it would be important to ensure inclusion of the views of the women themselves in the design , priority-setting and development of health services . Saint Kitts and Nevis Statistics on Saint Kitts and Nevis are not easy to come by and the country report had not yet been posted on the Internet at the time the present report was written . Because of this , the following analysis is limited in detail and only a few issues are raised for the consideration of the Committee . General Saint Kitts and Nevis has a total population of 38,000 people27 and , during the decade 1990-2000 , had a negative population growth rate ( -0.8).28 Sixty-six per cent of the people live in rural areas.29 Life expectancy at birth is 72 years for women and 66.1 years for men.30 The proportion of urban population to total population ( 34.1
  per cent)31 is considerably lower in Saint Kitts and Nevis than in the rest of the Caribbean ( 63 per cent).32 According to the World Bank , Saint Kitts and Nevis falls in the upper middle income category with an annual GNP per capita of $6,420.33 Children and adolescents Figures on both infant and child mortality reflect the biological advantage of girls over boys . The infant mortality rate is 17 for girls and 23 for boys ; 34 the child mortality rate is 22 for girls and 25 for boys.35 Although , unfortunately , the data is not disaggregated for boys and girls , immunization rates are good : 99 to 100 per cent of the children are fully immunized by one year of age.36 However , notwithstanding the relatively good economic circumstances of the country , nutritional levels , at least during the five-year period 1995-2000 , were not good in either absolute or relative terms ; they ran the same or higher than world levels and markedly higher than regional levels . Table 2 Prevalence o
 f child malnutrition in Saint Kitts and Nevis ( Percentage of children under five years of age)37 Such high malnutrition levels must be a matter of concern . Comment The Committee may wish to encourage the State party to investigate the causes of malnutrition , giving particular attention to establishing if and what kind of differences there may be in the nutritional status of boys and girls and whether there are any discriminatory feeding patterns which negatively influence a girl 's chances of being well nourished . Women Little data on specific health conditions of women in Saint Kitts and Nevis are generally published . There are , however , data on maternal mortality indicating the 1998 rate was 130 per 100,000 live births.38 Comment In global terms , the maternal mortality rate is comparatively high . The Committee may therefore wish to enquire what factors led to this rate and the State party may be encouraged to take action to improve available services and information , inc
 luding through community-based health services , with a view to improving this situation for women . The Committee may also wish to call attention to the special needs of older women and those with disabilities . Suriname General The population of Suriname totals 417,000 people,39 almost equally divided between men and women ( women = 50.36 per cent ) . The annual population growth rate is 0.4 per cent.40 Thirty-one per cent of the population is below the age of 15 years ( 127,200 people).41 Nearly one quarter of the population ( 23 per cent ) is made up of adolescents from 10 to 19 years of age.42 There is some difference in life expectancy at birth between women and men , with women expecting longer lives ( 73.5 years ) than men ( 68 years).43 Nearly three quarters of the people of Suriname live in urban areas ( 74.2 per cent).44 According to World Bank classifications , Suriname fell in the lower middle income group of nations in 2000.45 Although there is data in the public domai
 n about health in Suriname , including information specifically about women 's health , much of it refers to studies in a particular institution or locality . Much of it was also found to be six or more years old . For these reasons , it is hard to draw informed , overarching conclusions about the status , trends and challenges facing the people of Suriname in the field of women 's health . These notes attempt to suggest areas of interest or concern but , of necessity , are limited in scope . Children and adolescents As is true in most of the world , infant mortality rates , the proportion of children who die before they reach the age of one , are different between girls and boys , with the situation being better for girls . The rate for girls is 19 per thousand live births ; for boys , it is 27 per thousand live births.46 This is also the case for child mortality rates ( the proportion of children who die before reaching the age of 5 per thousand live births ) . The rate was 27 for
  girls and 29 for boys,47 an improvement over 1997 when they were considerably higher ( respectively , 39 and boys 51).48 Unfortunately , good , comprehensive , disaggregated data on child nutrition seemed unavailable for Suriname . Data on education shows equity between girls and boys , with girls making up 49.5 per cent of the primary school students ( equal to their percentage of that age population).49 General concerns about children have been well covered in the Suriname report to the Committee on the Rights of the Child and in the concluding observations of the Committee.50 Health-related issues of particular concern , which are rooted in discrimination , are violence and sexual exploitation of girls , beginning with young children and continuing into adolescence . While some variation exists among different ethnic communities , there is no question that HIV/AIDS is a serious problem that has an impact on children and older adolescents . The female-male distribution of HIV , p
 articularly among adolescents and young adults , highlights the difficulty girls have protecting themselves . It is estimated ( 2001 ) that among the population from 15 to 24 years of age , 0.79 per cent of boys are HIV positive , while the corresponding figure for girls is 1.33 per cent , nearly twice that of boys.51 At the end of 1999 , there were also 110 children below 15 years of age living with AIDS ( sex not reported).52 Furthermore , 352 children had been orphaned by AIDS.53 Although recent figures were hard to come by , it is likely that smoking is a serious and growing problem among girls . Already in 1987-1988 , smoking prevalence among young people from 10 to 19 years of age was estimated to be 12 per cent for girls and 36 per cent for boys.54 If Suriname has followed patterns common elsewhere , the advertising industry will have actively targeted young people and those figures are likely to have grown significantly . Knowing that the process of addiction grows more quic
 kly among young girls than boys , one must assume many of these girls will be lifelong smokers , with negative health effects accumulating throughout later years of life and , should the girls become pregnant , smoking will have a negative impact on the health of the baby they will bear . Comment Three issues of particular importance which would benefit from the Committee 's attention are domestic violence , with particular attention to abuse , including sexual abuse of girls ; information , care and service related to HIV among adolescents , particularly adolescent girls ; and smoking . Activities are needed to encourage , particularly among younger adolescents , empowerment of girls , more sexual responsibility among boys and more equitable and mutually respectful relations . Such activity would have a positive impact in both the short and long run , including a positive intergenerational impact , as children are less likely to be raised in families affected by violence or the HIV
  pandemic . Additionally , the Committee may wish to encourage the State party to attempt regular nationwide updates of disaggregated data on issues such as nutrition , smoking and domestic violence as the basis for gender and situation analysis leading to interventions to promote and protect the health of young people . Women Access to health-care services among minority and poor women is limited , a negative factor related to women 's health . Problems of access appear to stem from poverty , uneven distribution of service ( urban and interior areas ) and population . Gender-based violence is a serious problem , one which has been noted for at least the last decade , but progress in addressing it has been limited . A study carried out in 1990 found that 95 per cent of all sexual offences registered by the Emergency Department of the Academic Hospital were women : 20 per cent of the victims were below 10 years of age.55 While the Government has taken modest steps to improve the offi
 cial response to domestic violence , including some training for police , the availability of a social worker in some places , the coverage has been limited and girls and women of all ages continue to be subject to abuse . The women 's movement has also been active and by the late 1990s some progress was reported . A police study ( 1996-1997 ) indicated some decline in violence,56 although women were still victims in 76.7 per cent of the incidents of domestic violence reported in 1997.57 Debate , nonetheless , continues about the appropriateness of public policy entering an area of private relations such as domestic violence . This may slow official action in this field and permit continuing violence against women . HIV is an increasingly serious problem in Suriname . Problems of poverty and changes in some norms and values have contributed to an increase in prostitution.58 Moreover , men 's continuing decisive power over the use of condoms has meant that activity to control and tre
 at sexually transmitted infections , including HIV , have had limited benefits for women.59 In the late 1990s , the Dermatological Service indicated 1 in 5 street sex workers tested to be HIV positive.60 The overall infection rate of adults was 1.26 per cent in 1999,61 considerably higher than the regional ( Latin America ) rate of 0.5 per cent in 2001.62 Exhibiting a pattern common to many parts of the world , among younger people from 18 to 25 years of age , the majority of those who are HIV positive are female , while among those over 25 years of age , more males are infected.63 By the end of 1999 , women made up 33 per cent of all people aged 15 to 49 living with AIDS.64 The maternal mortality ratio for Suriname was estimated to be 230 per 100,000 live births for the early 1990s , well below the global average of 400 per 100,000 live births.65 A preliminary WHO analysis for 2000 suggests it is now much lower , around 112 per 100,000 live births . Nonetheless , given the fact tha
 t 98 per cent of deliveries are attended by trained health personnel,66 further investigation would be in order as to the distribution and causes of maternal mortality as well as options for affordable , acceptable interventions to reduce that figure . Comment The multi-ethnic make-up of Suriname and unequal distribution of population complicate the implementation of public policy and the achievement of equity among women with regard to access and utilization of public services . Inequities in power relations between men and women also make maintenance of good health difficult for some women who are in relationships which place their sexual , mental and physical health at risk and leave them little recourse . The Committee may wish , therefore , to encourage the State party to give particular attention to the development and implementation of stronger public policy in these areas as well as to issues of equity in the accessibility and utilization of public services . Data and report
 s on HIV infections and domestic violence indicate continuing , unresolved challenges related to gender relations . It is regrettable that no recent data has been found on smoking among adolescents or adults . This has long-term health impacts and advertisers often capitalize on and reinforce negative gender stereotypes to promote their product . The Committee may therefore wish to encourage the State party to investigate and take action to monitor and control rigorously access to and use of smoking material . Nutrition and mental health are two other fields of considerable impact related to women 's health and well-being on which little current data was found . Older women Women over 60 years of age make up 9 per cent of the female population of Suriname.67 As elsewhere in the world , they outnumber men of the same age by a considerable number \u2014 there are 127 women for every 100 men 60 years of age or older and there are 132 women for every 100 men 80 years of age or older.68 Whi
 le most people have access to some form of health insurance , the elderly and the poor have suffered most from the country 's economic problems , which have eroded the availability of services . Suriname is a country which has experienced a considerable outflow of population , especially those persons of productive age , and benefits from remittances and , sometimes , medicines sent home by family members employed abroad . Older women who do not have such networks , particularly those suffering from chronic diseases who need regular medical care or treatment , are particularly disadvantaged.69 Comment The State party is to be congratulated on its efforts to have insurance coverage for its people . However , the Committee may wish to encourage special efforts in times of hardship to identify people , particularly older women and the poor , who may need priority support and attention . 
\ No newline at end of file


[14/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/22syndicate_the-end-of-the-russia-china-debate.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/22syndicate_the-end-of-the-russia-china-debate.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/22syndicate_the-end-of-the-russia-china-debate.txt
new file mode 100644
index 0000000..2b92276
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/22syndicate_the-end-of-the-russia-china-debate.txt
@@ -0,0 +1,2 @@
+
+The End of the Russia-China Debate Ten years ago , comparing the reform processes in China and Russia was an intellectual fashion . Was it preferable to start with economics \u2013 try and get rich , quick , but don\u2019t rock the boat politically \u2013 in the manner of the Chinese ? Or was it better to start with politics \u2013 recover liberty and prosperity may follow \u2013 which seemed to be Russia\u2019s path under Mikhail Gorbachev and Boris Yeltsin ? Today , a new comparative debate has started . This time the subject is no longer Russia versus China , because Russia has long ceased to be a point of comparison . Instead , the new comparative debate involves Asia\u2019s two new economic , demographic , and political giants , China and India . China\u2019s annual economic growth has been roughly 8-9 % for the last 26 years ; India has recorded similar rates for the last decade . In the \u201c flat world \u201d of globalization \u2013 to borrow Thomas Friedman\u2019s powerful metaphor \u2013 it seems that Russia n
 o longer has a place . Of course , Russia is still the second-largest nuclear power in the world , and , as one of the world\u2019s leading exporters of oil and gas , it benefits from today\u2019s high energy prices . But Russia\u2019s population is disappearing before our eyes . With average male life expectancy just 57 years , the country is losing close to 800,000 people annually . Indeed , Russia is more a fragile oil-producing state than a modernizing economic giant . To put it bluntly , Russia is no longer in the same category as China . Whereas the \u201c Middle Kingdom \u201d is proudly regaining its former global status after centuries of decline , Russia is defiantly trying to resurrect its former imperial status , but in a manner that appears doomed to fail . Russia has clearly taken giant steps in the wrong direction , at a time when China has taken steps , however small , in the right direction . When you meet today\u2019s \u201c new Russian Nomenklatura , \u201d you experience a sudden feelin
 g of being 20 years younger , awash as they are in nostalgia for Cold War posturing . By contrast , seen from afar , in their new suits , you might think that Chinese economic elites are Japanese . Where Russia represents a return to the past , one sees in China an opening , however ambiguous , to the future . Of course , some bias is involved here . As a European , I and people like me almost instinctively expect more from Russia . It is , after all , a European nation in cultural , if not political terms , whereas progress in China will not be measured by the introduction of Western-style democracy , but eventually , one hopes , by Singapore-style rule of law . The diverse paths followed by Russia and China may be explained in part by how the two peoples perceive themselves . The Chinese are comforted in their self-image by the world\u2019s combination of admiration for their dynamism , greed for the market they constitute , and apprehension for the competition they represent . Russi
 ans , by contrast , seem to be animated by a dark form of narcissism . They do not find anything to be proud of in the eyes of others . They are respected for what they control \u2013 the Soviet legacy of nuclear arms and \u201c Christian energy resources , \u201d to quote Vladimir Putin\u2019s bizarre remark on his first official trip to Paris \u2013 but not for their economic performance or their essence . China and Russia tend to relate to their respective pasts and futures in very different ways \u2013 with self-confidence in China\u2019s case , with self-diffidence in the case of Russia . Chinese elites are convinced that time is working in their favor , and that it is only natural that China should regain its rank amongst the world leading powers , perhaps even emerging on top one day . Indeed , their serene patience stands in stark contrast to the anxious reticence of Russian leaders , who have yet to surmount the humiliation that Russia suffered as a result of the Soviet Union\u2019s disintegration
  at the end of the Cold War . Russia may be experiencing a global \u201c restoration \u201d phase , but in politics and economics , and also with respect to its empire , restoration is headed in the wrong direction . With Putin\u2019s current clampdown on civil society , re-nationalization of key segments of the economy , failure to develop any political approach to resolving the conflict in Chechnya , and cultivation of imperial nostalgia , Russia is killing its only chance to matter in the future . Yet there is no reason for China to declare victory . The gap between the respective qualities of China\u2019s economic and scientific elites , on the one hand , and its ruling political elites , on the other , is simply too monumental \u2013 and still growing \u2013 for stability to be taken for granted . One senses in China the birth pangs of civil society , making the introduction of the rule of law increasingly urgent . Without political reforms , China\u2019s confidence in herself will quickly turn int
 o disillusion , or even delusion . If that happens , the Russia-China debate might be revived , this time as a comparison of competitive decadence . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/23syndicate_the-making-of-china-s-trade-deficit.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/23syndicate_the-making-of-china-s-trade-deficit.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/23syndicate_the-making-of-china-s-trade-deficit.txt
new file mode 100644
index 0000000..24980d1
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/23syndicate_the-making-of-china-s-trade-deficit.txt
@@ -0,0 +1,2 @@
+
+The Making of China\u2019s Trade Deficit BEIJING \u2013 China registered a monthly trade deficit of $7.2 billion in March 2010 , its first since April 2004. And yet , at around the same time , the United States Congress issued its loudest call ever to classify China as an exchange-rate manipulator , accusing Chinese leaders of maintaining the renminbi\u2019s peg to the dollar in order to guarantee a permanent bilateral trade surplus . China\u2019s March trade deficit indicates , first of all , that it is incorrect to claim that Chinese economic growth depends mainly on exports . Exports are an important part of the Chinese economy , and any global market fluctuation or external shock will certainly have an impact on overall growth . But , like any other large economy , China\u2019s economy is driven by domestic consumption and investment . Indeed , China\u2019s exports fell by 16 % year on year in 2009 , owing to the global financial crisis and recession . Nevertheless , annual GDP grew by 8.7 % , th
 anks to 16.9 % growth in consumption ( measured by gross sale of consumer goods ) and a 33.3 % surge in fixed-investment demand . Moreover , although China\u2019s \u201c trade dependency \u201d is now reckoned to be 70 % of GDP , that figure is greatly distorted by the fact that Chinese exports require massive imports of materials and parts . The net value added of total Chinese foreign trade accounts for only about 15 % of GDP . Thus , net exports contributed 10.8 % to China\u2019s overall GDP growth rate , or only about 1.1 percentage point of 9 % growth in 2008. Compare that figure to Germany , where net exports accounted for 64 % of growth in 2008. Similarly , the figure was 33 % in Japan , 28.6 % in Korea , and 20 % in the Philippines . Clearly , China is nothing special in this regard . To be sure , China\u2019s domestic consumption is not as high as it should be , standing at 49 % of GDP in 2008 , with household consumption accounting for only 35 % . Such figures have led many observers to 
 believe that overall domestic demand must be low , leaving China dependent on external markets for growth . But domestic demand , which determines imports , consists not only of consumption , but also of fixed-asset investment . Indeed , rapid growth in investment may translate into high import growth and trade deficits . That is exactly what is happening in China now . Some people may argue that investment growth without consumption growth will result in overcapacity and eventually lead to recession . Perhaps . But we need to remind ourselves that housing investment accounts for about 30 % of China\u2019s total fixed investment , with much of the rest directed toward infrastructure \u2013 that is , long-term , durable public infrastructure investments \u2013 including subways , railways , highways , urban public facilities , and the national water system . Moreover , one can easily imagine that import demand would soar further if the US and the European Union lifted their bans on exports of
  high-tech products to China . In that case , the trade deficit recorded in March could be at least 40 % higher . The renminbi\u2019s exchange rate , then , is really a secondary factor in China\u2019s external account . Put another way , the global imbalance could be corrected more efficiently by addressing other , more fundamental factors . The fundamental factors underlying the US external imbalance are large fiscal deficits and low household savings , owing to excessive financial leverage . The fundamental factors on the Chinese side are high corporate and household savings , together with some distortion of resource/utility prices . Indeed , the current situation indicates that a significant adjustment in exchange rates may not be needed at all in order to redress global imbalances . If that is true , and China shouldn\u2019t worry about the renminbi\u2019s exchange rate , should it worry about its economy overheating ? After all , its previous trade deficits in the era of reform \u2013 such 
 as in 1992-1996 and 2003-2004 \u2013 all occurred at times of overheating . But there are differences between now and those earlier periods . For example , when rapid investment growth in 2004 led to overheating , world markets were booming . At that time , both domestic investment and exports required immediate tightening . Today , by contrast , although domestic investment is growing very strongly , external demand has not recovered to its previous levels . The result is the March trade deficit , caused mainly by exceptionally high annual import growth ( 65 % ) coupled with relatively low export growth , which reached a nominally impressive 24 % only because of the sharp decline recorded in the base period . Such a single-factor situation is easier to deal with than the double-factor situation of 2004 , and because the high investment demand has been mainly stimulus-related this time , policymakers can handle it in a more timely fashion if they perceive a problem . That said , the ra
 tio of capital formation does require careful monitoring . The last time China saw such high growth in domestic investment , the savings rate was not as high as it is now . The problem currently is that a trade deficit has emerged at a time when the national saving rate is as high as 51 % . That means that investment is extremely high \u2013 and that , despite the high share of infrastructure investment , there is an urgent need to manage the potential risks . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/24syndicate_the-return-of-franco-german-leadership.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/24syndicate_the-return-of-franco-german-leadership.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/24syndicate_the-return-of-franco-german-leadership.txt
new file mode 100644
index 0000000..6653852
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/24syndicate_the-return-of-franco-german-leadership.txt
@@ -0,0 +1,2 @@
+
+The Return of Franco-German Leadership PARIS \u2013 Whoever wins September\u2019s parliamentary election in Germany , the time has come once again for a major Franco-German initiative . Regardless of their economic conditions or their confidence \u2013 or lack of it \u2013 in each other , France and Germany are more than ever jointly responsible for the future , if not the very survival , of the European project . Are there alternatives to Franco-German leadership of the European Union ? Joining Great Britain with them in a Club of Three would be a good idea , but it is out of the question nowadays . Britain has largely excluded itself from any leadership role in Europe . Gordon Brown is barely surviving as prime minister , and the Conservatives , whose return to power in the next year is almost certain , are as provincially euro-skeptic as ever , if not more so . Europe simply cannot count on the British , at least for a while . The idea of a Club of Six , floated by Nicolas Sarkozy early in h
 is presidency , was always abstract and is now untenable . Given Silvio Berlusconi\u2019s sexcapades , the Italy that he leads cannot be taken seriously , while Spain is out of the running for an EU leadership role , owing to its dire economic conditions . As for Poland , although the bumbling Kaczynski \u201c twins \u201d have been removed from power , the country\u2019s fixation on security in its immediate neighborhood is incompatible with true European leadership . Since the other 21 EU members never liked the idea of a Club of Six to begin with , it is just as well that such a vision has been buried , probably forever . So where but France and Germany can Europe turn for leadership ? A positive referendum result on the Lisbon Treaty in Ireland in October would be a necessary but insufficient condition to jump-start an institutional re-launch of the EU . Above all , the EU needs political will and direction . Only Germany and France , acting together , can convey to Europe\u2019s citizens and 
 to the world the sense that the EU is at long last waking up to today\u2019s global realities . Of course , it is impossible to underestimate the combination of suspicion , tension , and exasperation that characterizes Franco-German relations lately . To a large extent , Germany has become a \u201c second France \u201d in Europe , at a time when France is more French than ever . And they are not only putting their respective nationalisms first . They disagree on fundamentals \u2013 most of all , about how to surmount the economic crisis . But the two giants of Europe can agree to disagree about the virtues of German-style budgetary rigor or French-style fiscal stimulus as long as they don\u2019t insult each other , and , more importantly , as long as they compensate for their philosophical differences with a well publicized program of joint initiatives on key subjects . As long as each remains convinced that no alternative to cooperation exists within the EU , and that European cooperation remains
  a priority for both , it should not be overly difficult to restore their leadership . After all , France and Germany are closer to each other on many key topics than they have been for a long time . With the return of France to NATO\u2019s integrated military structure , the two countries are on the same \u201c Atlantic \u201d wavelength for the first time since 1966. Despite both sides \u2019 deep reservations about the mission in Afghanistan , they are clearly in the same boat , even if French troops , being closer to British troops in terms of combat , are more vulnerable than the Germans . And both countries would probably subscribe to the following formulation : \u201c Turkey\u2019s future is with Europe , but not necessarily in the European Union , at least in the foreseeable future . \u201d The fundamental question about how to deal with Russia remains a divisive issue , however . France and Germany have different sensitivities on the subject , which is both natural and inevitable , as these dif
 ferences reflect both geography and history . Germany is not only much closer physically to Russia ; it is also much more dependent on Russia in terms of energy security . France must not delude itself : Germany is not about to convert to nuclear energy to reduce its reliance on Russian oil and gas . Yet Germany also must realize that Russia\u2019s negative evolution has consequences that Germans cannot escape . A spectacular Franco-German security initiative following the election in Germany , accompanied by a joint message to the Kremlin , would also have the benefit of sending a message to the rest of the EU , particularly to its V�clav Klauses : \u201c If you decide to paralyze the Union through stubborn ill will , you will only end up excluding yourselves , rather than dictating Europe\u2019s fate . \u201d France and Germany cannot move Europe alone , but Europe without them cannot move at all . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/25syndicate_the-transatlantic-index.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/25syndicate_the-transatlantic-index.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/25syndicate_the-transatlantic-index.txt
new file mode 100644
index 0000000..1af22cb
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/25syndicate_the-transatlantic-index.txt
@@ -0,0 +1,2 @@
+
+The Transatlantic Index How should we assess the state of transatlantic relations nowadays ? With a nod to Wall Street , we can say that the Alliance is up , Europe is flat , and the United States is clearly down . The Alliance is \u201c up \u201d for one key reason : the warming of France\u2019s relations with the US following Nicolas Sarkozy\u2019s election as French president . For the first time since Charles de Gaulle established the Fifth Republic , France\u2019s priority is no longer to live in opposition to the US . The signs of this shift are profound , even spectacular . From a toughening of France\u2019s position on Iran to a real warming of relations with Israel , not to mention symbolic gestures like Sarkozy\u2019s summer vacation in America , or Foreign Minister Bernard Kouchner\u2019s arrival in Baghdad , this is a New France , one seriously considering a return to NATO\u2019s integrated military structure . France\u2019s shift is the result of both political calculus and deeply felt emotion . For
  Sarkozy , the French are not anti-Americans , but simply anti-Bush . In his willingness to break with the past \u2013 in particular with Jacques Chirac\u2019s legacy \u2013 and in adding a global spin to his \u201c mandate for change , \u201d Sarkozy is paving the way for the post-Bush America that will soon be here . He knows that the \u201c return of France \u201d as an influential actor in Europe presupposes a France that is closer to America , which will automatically bring France closer to Germany and Great-Britain , not to mention Poland and Italy . Emotionally , too , \u201c Sarkozy the American , Sarkozy the doer , \u201d who wants to be judged by his actions , tends to see himself as a Gallic incarnation of the American dream . He is the son of immigrants , the outsider whose rise to the top is living proof of French openness . In the US , too , the attitude towards the alliance with Europe has changed mightily . The failure in Iraq , the risk of an \u201c Iraqization \u201d of Afghanistan , and Russia\u2019
 s newfound assertiveness have moved America from the arrogant indifference that characterized most of Bush\u2019s presidency to a rediscovery of the Alliance\u2019s value . America needs allies , and is not preoccupied by their potential independent strength . As one top American diplomat put it , \u201c The last worry I have when I wake up at three in the morning is that Europe is becoming too strong . \u201d But rapprochement between France and the US should not hide other realities . First , in Afghanistan , NATO is in danger of suffering its first military defeat . Where will the new troops that are needed come from ? And a key member of the Alliance , Turkey , may be about to embark in a dangerous military adventure in Kurdish northern Iraq . Second , one cannot speak of NATO without noting that the Alliance\u2019s twin pillars \u2013 Europe and the US \u2013 are not in great shape . If Europe is \u201c flat , \u201d one reason is Sarkozy . He may be encouraging news for the Alliance , his European polic
 y is a source of worry for Europe . While Sarkozy claims that Europe is his priority , his methods seem to contradict his intentions . The difficult personal chemistry he has with German Chancellor Angela Merkel is reinforced by his constant attacks on the European Central Bank and its president , Jean-Claude Trichet . One cannot simultaneously defend the Alliance in the name of Europe and weaken Europe with fits of populism and economic nationalism . Europe\u2019s three key new leaders \u2013 Britain\u2019s Gordon Brown , Merkel , and Sarkozy \u2013 all belong to a generation that is no longer emotionally moved by the project of European integration . Their links with Europe \u2013 if they exist at all \u2013 are at best rational , not emotional . But will cool rationality be enough to create a European security pillar within the context of the Alliance ? As for America , it is \u201c down \u201d in terms of both \u201c soft \u201d ideological power and \u201c hard \u201d military power . The US remains by far the wo
 rld\u2019s strongest country , particularly in military terms , but it is confronted with fundamental questions about the use and utility of force at a time when power is relative . Today , Americans and Europeans alike must demonstrate modesty . Unlike when the Alliance was created , a multi-polar world system has taken shape , in which the West\u2019s demographic and economic share has fallen , and in which it must now compete with successful authoritarian models such as China or even Russia . In this new context , solidarity is as crucial as ever . Opinion about the US remains largely negative in Europe , and will remain so at least until the US presidential election in 2008. Likewise , US perceptions of France and of Europe are only slowly improving . It is only by respecting our common values and not exacerbating our differences that the West will be able to reinvent itself . The Alliance may be \u201c up , \u201d but its outlook remains uncertain . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/26syndicate_will-russia-save-the-west-.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/26syndicate_will-russia-save-the-west-.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/26syndicate_will-russia-save-the-west-.txt
new file mode 100644
index 0000000..62008c5
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/26syndicate_will-russia-save-the-west-.txt
@@ -0,0 +1,2 @@
+
+Will Russia Save the West ? MOSCOW \u2013 Rapid changes in the global economy and international politics are raising , once more , an eternal issue in Russia : the country\u2019s relations with Europe , and with the Euro-Atlantic region as a whole . Of course , Russia partly belongs to this region . Yet it cannot and does not want to join the West wholeheartedly \u2013 at least not yet . Meanwhile , this choice looks very different now compared to just a few years ago . It is becoming obvious that the Euro-Atlantic world , whose economic and political model seemed so triumphant 20 years ago , is now lagging somewhat behind China and other Asian countries . So is Russia , where , despite encouraging talk about innovation-based development , the economy continues to de-modernize as corruption has been allowed to metastasize , and as the country relies increasingly on its natural-resource wealth . Indeed , it is Asia that has turned out to be the true winner of the Cold War . These rising power
 s raise problems regarding Russia\u2019s geo-strategic choices . For the first time in decades , the values gap between Russia and the EU appears to be increasing . Europe is overcoming state nationalism , while Russia is building a nation state . Broken by history and not wishing to be ravaged by war again , Europeans have embraced compromise and renounced the direct use of force in international relations . Russians , on the other hand , emphasize their \u201c hard power , \u201d including military force , because they know that they live in a dangerous world and have no one to hide behind . And , because of the country\u2019s comparative lack of \u201c soft power \u201d \u2013 social , cultural , and economic attractiveness \u2013 it stands ready to use the competitive advantages ( i. e. , its resource wealth ) available to it . Internal political developments in Russia are also pushing the country in a different direction from the West . Quite simply , Russia is moving away from democracy . This emergi
 ng values gap is not an insurmountable obstacle to geostrategic rapprochement . But , coupled with mutual irritation , which is particularly strong in Russia , closing the gap is becoming much harder . For , while Russia \u2019 s elite never considered itself defeated in the Cold War , the West essentially treated Russia as a defeated country \u2013 an attitude symbolized by NATO\u2019s eastward expansion , which laid a deep foundation for ongoing tension . It was only after the West encountered an armed rebuff in South Ossetia that NATO expansion was stopped in its tracks . Yet NATO has not given up on further enlargement . NATO expansion is nothing more than the extension of its zone of influence \u2013 and in the most sensitive , military-political sphere . And yet the West\u2019s unwillingness to abandon that effort is coupled with a repeated refusal to recognize Russia\u2019s right to have its own zone of interest . So NATO expansion has left the Cold War unfinished . The ideological and militar
 y confrontation that underlay it is gone , but the geopolitical rivalry that it entailed has returned to the fore . Thus , the old mentality survived on both sides . Energy debates are another example of this . Non-Russian Europe should thank the Almighty for the presence of energy-rich Russia at its borders , while Russia should be thankful for having such wealthy customers . But the natural differences in the interests of energy consumers and producers have been given a political/security twist \u2013 witness the discussions about an \u201c Energy NATO . \u201d F aced with the impossibility of advantageous accession to Euro-Atlantic institutions , Russia is drifting fast towards alignment with China \u2013 a \u201c younger brother , \u201d though a respected one . Russia\u2019s \u201c Asian choice \u201d of today is not the same as the Slavophile/Eurasian choice of the past . On the surface , it looks like a choice in favor of a rapidly rising civilization . But the current estrangement from Europe \u2013 the 
 cradle of Russian civilization and modernization \u2013 threatens Russia\u2019s identity and will increase its geostrategic risks in the future . Europe does not benefit from this estrangement either . It will continue to move towards beautiful decay \u2013 Venice writ large . The United States also loses . Without Russia , which will remain the world\u2019s third strongest power for the foreseeable future , it is impossible to solve the key problems of international security . The current Euro-Atlantic security architecture seems to suit the majority of Americans and Europeans , though it is becoming increasingly fragile and counterproductive . So Russia will struggle to create a new architecture largely on its own \u2013 whether through a new treaty on collective European security , or even through its accession to NATO . This is not only in Russian political and civilizational interest , but it also reflects our duty to the entire community of Euro-Atlantic nations , which is being weakened by 
 the \u201c unfinished Cold War . \u201d The idea of a \u201c Union of Europe \u201d between Russia and the EU should be put on the long-term agenda . That Union should be based on a common human , economic , and energy space . The combination of a new security arrangement for the Euro-Atlantic community and the establishment of the Union of Europe could arrest the decline in the international weight of the West . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/2syndicate_china-and-russia-in-the-new-world-disorder.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/2syndicate_china-and-russia-in-the-new-world-disorder.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/2syndicate_china-and-russia-in-the-new-world-disorder.txt
new file mode 100644
index 0000000..09a1f81
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/2syndicate_china-and-russia-in-the-new-world-disorder.txt
@@ -0,0 +1,2 @@
+
+China and Russia in the New World Disorder Can Kosovo achieve independence without the tacit consent of Russia , and can there be a humanitarian and political solution to the tragedy in Darfur without the active goodwill of China ? The two crises have nothing in common , but their resolution will depend in large part on whether these two permanent members of the United Nations Security Council use their veto power . Comparing the respective abilities of Russia and China to block key international initiatives makes no sense in itself , but it does constitute a useful tool for understanding the transformation of the international system that is now taking place as a direct consequence of the relative decline of America\u2019s global power . From that standpoint , the deepening of chaos in the Middle East poses both opportunities and risks for Russia and China , which may force them to define the roles they want to play and the images they want to project in the world . The key question i
 s this : Is Russia taking giant steps in the \u201c wrong direction \u201d while China is taking \u201c minuscule \u201d steps in the \u201c right direction \u201d ? Superficially , Russia and China may give the impression that they are pursuing the same path when they both proclaim with pride that they are \u201c back \u201d on the world stage . But this boast means different things for each country . For China , a deeply self-confident country , to be \u201c back \u201d simply means regaining the country\u2019s historical centrality in the world after an absence of more than two centuries . After all , at the end of the eighteenth century , China became the world\u2019s first producer of manufactured goods , and it perceives itself as a center of civilization unequalled by any other in Asia , if not the world . China\u2019s renewed self-confidence is based on its remarkable economic prowess , which is derived not from natural resources , but from productivity and creativity . Whatever the huge political , social , and 
 economic tensions may exist , there is a \u201c feel good \u201d factor in China , a sense of progress , with the 2008 Olympics in Beijing figuring as the symbolic moment that will proclaim to the world the scale of the country\u2019s achievements . Above all , with the exception of the Taiwan issue , China is a satisfied status quo power when it comes to the evolution of the international system \u2013 a patient actor that finds it perfectly legitimate to behave and to be seen as the world\u2019s number two power . By contrast , the Russians remain insecure about their status in the world . Russia\u2019s explosive \u201c revisionist \u201d behavior on the eve of the recent G8 summit is an indication of the Kremlin\u2019s \u201c unsatisfied \u201d nature . Because they know they are less potent , particularly in demographic and economic terms , Russians feel they have to do \u201c more . \u201d For them , to say \u201c Russia is back \u201d means that the humiliating Yeltsin years are over , and that they now must be treated a
 s equals , particularly by the United States . Russians are nostalgic not for the Cold War as such , but for the international status they lost when it ended . Now that America is no longer a \u201c hyperpower \u201d with no strategic challengers , Russia has reasserted its status as a \u201c superpower , \u201d a claim that is not necessarily supported by reality . Unlike the Chinese , the Russians do not create economic wealth , but merely exploit their energy and mineral resources . Moreover , unlike the Chinese , they have not always been confident of their position in the world . Torn between Europe and Asia in cultural and political terms , victimized by a dark , narcissistic instinct that pervades their reading of their past and their visions of the future , it should surprise no one that Russia is now behaving like a \u201c revisionist \u201d power . Unsatisfied with their inner identity , it is only natural that Russians should demand changes that make them feel more secure and proud . Less 
 than 20 years ago , the Czech Republic and Poland were part of their sphere of influence , so Russians understandably cannot accept the US unilaterally implanting its security system there . Of course , in their respective judgments on Russia and China , the West \u2013 and Europeans in particular \u2013 may be demonstrating selective emotions . \u201c We \u201d tend to be less demanding of China than of Russia , because we tend to see Russia as \u201c European \u201d ( at least culturally ) . As a result , the culture of physical violence and verbal provocation that is gaining ground in Putin\u2019s Russia is deeply disturbing , whereas we tend to judge Chinese misdeeds with a greater sense of distance , if not indifference . But the differences between Russia and China today may prove to be less significant tomorrow if the deterioration in the Middle East imposes a sense of collective responsibility on all five permanent members of the UN Security Council . It is one thing for Russia and China to expl
 oit America\u2019s growing difficulties from Gaza to Kabul ; but it is quite another if the situation deteriorates to the point of general destabilization in the region . Indeed , the Middle East\u2019s deepening problems may impose a sense of restraint in Russia and China by forcing them to calculate not in terms of their global \u201c nuisance value \u201d vis-�-vis the West , but in terms of their ability to make a positive and stabilizing contribution to the world order . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/3syndicate_china-japan-protests-islands-by-liah-greenfeld.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/3syndicate_china-japan-protests-islands-by-liah-greenfeld.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/3syndicate_china-japan-protests-islands-by-liah-greenfeld.txt
new file mode 100644
index 0000000..149f95b
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/3syndicate_china-japan-protests-islands-by-liah-greenfeld.txt
@@ -0,0 +1,2 @@
+
+The Roots of Chinese/Japanese Rivalry BOSTON \u2013 The anti-Japan protests that continue to roil China are just another indication of the rise of a potent Chinese nationalism . After a century slowly fomenting among Chinese intellectuals , national sentiment has captured and redefined the consciousness of the Chinese people during the last two decades of China\u2019s economic boom . This mass national consciousness launched the Chinese colossus into global competition to achieve an international status commensurate with the country\u2019s vast capacities and the Chinese people\u2019s conception of their country\u2019s rightful place in the world . Rapidly , visibly , and inevitably , China has risen . Indeed , our era will likely be remembered as the time when a new global order , with China at the helm , was born . Competitive national consciousness \u2013 the consciousness that one\u2019s individual dignity is inseparably tied to the prestige of one\u2019s \u201c people \u201d \u2013 worked its way into the mind
 s of China\u2019s best and brightest between 1895 and 1905. In 1895 , China was defeated by Japan , a tiny aggressor whom the Chinese dismissively called wa ( the dwarf ) . China was already accustomed to rapacious Western powers squabbling over its riches , but had remained self-confident in the knowledge of these powers \u2019 irrelevance . However , the assault from Japan , a speck of dust in its own backyard , shattered this self-assurance and was experienced as a shocking and intolerable humiliation . Japan\u2019s triumph in 1905 over \u201c the Great White Power , \u201d Russia , repaired the damage to China\u2019s sense of dignity . From the Chinese point of view , Russia was a formidable European power , one feared by other Western powers . Its defeat , therefore , was seen as a successful Asian challenge to the West , in which China , its intellectuals felt , was represented by Japan . Japan thus became the focus of Chinese attention . Gentlemen-scholars , who would reform and staff the Chin
 ese army and civil service in the early decades of the twentieth century , went to study in Japan . The Revolution of 1911 was inspired by the example of Japan\u2019s Meiji Restoration ; and , because early-twentieth-century Japan was stridently nationalistic , the new China that emerged from its image was constructed on nationalist principles as well . Thus Japan became the significant \u201c other \u201d for China , the model that was imitated and the anti-model that was resented . Chinese nationalism borrowed from Japan its concept of the nation , including the very word by which it was expressed ( kuoming , from the Japanese kokumin ) . The Kuomintang ( the Chinese Nationalist Movement ) was explicitly inspired by Japan and fueled by repeated Japanese aggression . Paradoxically , but not unexpectedly , Mao Zedong\u2019s struggle against the Kuomintang was inspired by anti-Japanese nationalism as well . As was the case virtually everywhere else , communism in China was nationalism incarnate 
 . Mao\u2019s speech on the establishment of the People\u2019s Republic plainly expressed the nationalist agenda behind it . Calling the nation \u201c communist \u201d assured the new People\u2019s Republic of China of the Soviet Union\u2019s support , which was viewed by Mao as more reliable than that of the United States . But neither the Russian nor the Chinese Communists were ever unclear about the nationalist nature of their respective projects . The upper echelons of the bureaucracy and intelligentsia in Russia and China were self-consciously nationalist and , throughout Communist rule , shrewdly pursued the supreme nationalist goal : prestige \u2013 the power , naked and otherwise , to impose the nation\u2019s will on others . But national consciousness , particularly in China , was limited to a narrow elite , leaving the masses almost untouched . This changed dramatically with the Chinese government\u2019s restoration of a capitalist economy . Much like in Germany in the 1840\u2019s , when the appeal to p
 rivate enterprise converted the entire middle class to nationalism , the explicit definition of economic power as the central pillar of China\u2019s greatness awakened ordinary Chinese to nationalism\u2019s appeal . Hundreds of millions now see themselves as sharing in the nation\u2019s dignity , and are eager to contribute to it , as well as to defend it from insult . Competition for prestige , even when the contest is economic , is not a purely rational undertaking . So it should be no surprise that old injuries are likely to resurface . Some Chinese , especially those who are not economically successful , harp bitterly on Japan\u2019s past depredations . Despite China\u2019s embrace of capitalism and Japanese investment , Japan remains China\u2019s reviled other . Indeed , a professor in Beijing told me not long ago , \u201c Two in every ten Chinese dislike the US , but nine in every ten hate Japan . \u201d For the West , there is a silver lining in this nationalist rivalry : neither China nor Japan is 
 a rogue state , and , so long as their quarrels do not lead to the use of unconventional weapons , we may treat the friction between them as an internal Asian quarrel . Japan , moreover , is likely to let today\u2019s passions over disputed islands in the East China Sea cool down , despite the anti-Japanese outbursts in Chinese cities . But the West \u2013 and the US in particular \u2013 is new to dignity games � la Chinois . If it gets carried away and presumes to talk down to the 5,000-year-old culture of the Sages , the West could become the next object of China\u2019s nationalist resentment . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/4syndicate_china-s-cyber-warriors.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/4syndicate_china-s-cyber-warriors.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/4syndicate_china-s-cyber-warriors.txt
new file mode 100644
index 0000000..c3267ed
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/4syndicate_china-s-cyber-warriors.txt
@@ -0,0 +1,2 @@
+
+China\u2019s Cyber-Warriors NEW DELHI \u2013 The world now accepts that protecting our atmosphere , hydrosphere , lithosphere , and biosphere \u2013 the \u201c global commons \u201d \u2013 is the responsibility of all countries . The same norm must apply to cyberspace , which is critical to our everyday life , economic well-being , and security . At a time when cyber attacks are increasing worldwide , US Secretary of State Hillary Clinton was right to declare that an attack on one nation\u2019s computer networks \u201c can be an attack on all . \u201d Indeed , the attacks are a reminder that , as a new part of the global commons , cyberspace already has come under threat . Cyberspace must be treated , along with outer space , international waters , and international airspace , as property held in common for the good of all . And , like ocean piracy and airplane hijacking , cyber-crime cannot be allowed to go unpunished if we are to safeguard our common assets and collective interests . Naming China among a ha
 ndful of countries that have stepped up Internet censorship , Clinton warned that \u201c a new information curtain is descending across much of the world . \u201d Her statement , with its allusion to the Cold War-era Iron Curtain , amounted to an implicit admission that the central assumption guiding US policy on China since the 1990\u2019s \u2013 that assisting China\u2019s economic rise would usher in greater political openness there \u2013 has gone awry . The strategy of using market forces and the Internet to open up a closed political system simply is not working . Indeed , the more economic power China has accumulated , the more adept it has become in extending censorship to cyberspace . If anything , China has proven that a country can blend control , coercion , and patronage to stymie the Internet\u2019s politically liberalizing elements . Through discreet but tough controls , Beijing pursues a policy of wai song , nei jin \u2013 relaxed on the outside , vigilant internally . Google is now crying f
 oul over \u201c a highly sophisticated and targeted attack on our corporate infrastructure originating from China . \u201d But , despite its corporate motto \u2013 \u201c Don\u2019t be evil \u201d \u2013 Google itself was instrumental in aiding online censorship in China , having custom-built a search engine that purges all references and Web sites that the Chinese government considers inappropriate . Now Google itself has become a victim of China\u2019s growing cyber prowess , in the same way that appeasement of Hitler boomeranged onto France and Britain . China deploys tens of thousands of \u201c cyber police \u201d to block Web sites , patrol cyber-cafes , monitor the use of cellular telephones , and track down Internet activists . But the threat to the new global commons comes not from what China does domestically . Rather , it comes from the way in which the know-how that China has gained in fashioning domestic cyber oversight is proving invaluable to it in its efforts to engage in cyber intrusion across it
 s frontiers . Canadian researchers have discovered a vast Chinese surveillance system called \u201c GhostNet , \u201d which can compromise computers in organizations abroad through booby-trapped e-mail messages that automatically scan and transfer documents to a digital storage facility in China . This is what happened when computers of the Tibetan government-in-exile in Dharamsala , India , were attacked last year . India\u2019s national security adviser recently complained that his office was targeted yet again by hackers . \u201c People seem to be fairly sure it was the Chinese , \u201d he said . Officials in Germany , Britain , and the US have acknowledged that hackers believed to be from China also have broken into their government and military networks . The state-sponsored transnational cyber threat is at two levels . The first is national , with the hackers largely interested in two objectives . One is to steal secrets and gain an asymmetrical advantage over another country . Cyber intrusi
 on in peacetime allows the prowler to read the content and understand the relative importance of different computer networks so that it knows what to disable in a conflict situation . The other objective is commercial : to pilfer intellectual property . The second level of cyber threat is against chosen individuals . The most common type of intrusion is an attempt to hack into e-mail accounts . The targets also can face Trojan-horse attacks by e-mail intended to breach their computers and allow the infiltrators to corrupt or transfer files remotely . To be sure , if a cyber attack is camouflaged , it is not easy to identify the country from which it originated . Through the use of so-called \u201c false-flag espionage \u201d and other methods , attacks can be routed through the computers of a third country . Just as some Chinese pharmaceutical firms exported to Africa spurious medicines with \u201c Made in India \u201d labels \u2013 a fact admitted by the Chinese government \u2013 some Chinese hacker
 s are known to have routed their cyber intrusion through computers in Russia , Iran , Cuba , and other countries . But , like their comrades in the pharmaceutical industry , such hackers tend to leave telltale signs . Then there are many cases in which the attacks have originated directly from China . It seems unlikely that these hackers , especially those engaged in cyber espionage , pilferage , and intimidation , are private individuals with no links to the Chinese government . It is more likely that they are tied to the People\u2019s Liberation Army . In war , this irregular contingent of hackers would become the vanguard behind which the PLA takes on the enemy . Systematic cyber attacks constitute a new frontier of asymmetrical warfare at a time when the world already confronts other unconventional threats , including transnational terrorism . With national security and prosperity now dependent on the safekeeping of cyberspace , cybercrime must be effectively countered as an intern
 ational priority . If not , cyberspace will become the new global-commons battlefield . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/5syndicate_china-s-gift-to-europe.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/5syndicate_china-s-gift-to-europe.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/5syndicate_china-s-gift-to-europe.txt
new file mode 100644
index 0000000..a881864
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/5syndicate_china-s-gift-to-europe.txt
@@ -0,0 +1,2 @@
+
+China\u2019s Gift to Europe Europeans are wrong to be angry with China because its currency peg to the US dollar has boosted the euro against most currencies on foreign exchange markets . On the contrary , they should view the currency peg as a valuable gift . In New York and other American cities , European shoppers are on a spree . They\u2019re enjoying first hand the euro\u2019s enhanced purchasing power abroad . The currency peg has also helped make the euro into an important reserve currency , rivaling the dollar . Investments from all over the world are flowing into Europe as a result . It was predictable that China\u2019s currency peg would have this effect . It forced China to accumulate huge amounts of dollars to stabilize its currency and gave the euro an extra boost as the dollar depreciated . To protect itself from the declining greenback , China and other countries are converting a portion of their accumulated dollar reserves into euros . The International Monetary Fund estimates t
 hat the euro\u2019s share of foreign exchange reserves rose to 26.4 percent in third quarter of 2007 from 25.5 percent in the second quarter of 2007 , and 24.4 percent in the third quarter of 2006. The euro\u2019s becoming a major reserve currency bestows important economic advantages on the euro-zone economy . Europeans should thank the Chinese currency peg for the part it is playing in this . Of course , with an augmented supply of euros , China can be expected to make substantial investments in Europe , which also is a good thing , especially since , in the United States at least , China has demonstrated a preference for investments in financial institutions rocked by the sub-prime crisis . Europe has plenty of those . So far , China\u2019s biggest European investment has been in the Belgian financial company , Fortis N. V. , where Ping An Insurance recently acquired a 4.2 % stake , making it the largest shareholder in the company . China Development Bank also made an investment in Barcla
 ys to bolster its bid for ABN AMRO which , though it failed , helped the Dutch bank get the full value for its assets . This is only a start . Merrill Lynch predicts that , as currency reserves climb in countries like China , state-run investment funds will grow to $7.9 trillion , from $1.9 trillion now . China\u2019s currency peg has also come in handy for Europe in its fight to keep inflation in check , while its monetary policy remains constrained by the sub-prime crisis . Experts estimate that , in terms of guarding against inflation , the euro\u2019s appreciation has been equivalent to about 35 basis points rise in the interest rate . The strong euro has helped insulate Europe from skyrocketing oil and commodity prices , which are priced in dollars on the world markets . The big downside of the currency peg for Europe is supposed to be exports . That\u2019s what sent French President Nicolas Sarkozy and other high-level European officials scurrying off to China last November\u2014and why m
 any ordinary Europeans continue to think of the Chinese as not playing by the rules . This is protectionist thinking par excellence . What makes exports so special ? Why are they more important than cheap imports or the bargains that European shoppers get in New York ? Should Europe give up all the advantages of the currency peg just because it makes European exports more expensive ? Of course not ! Note that it was French President Sarkozy who went to China to protest the currency peg , not German Chancellor Angela Merkel . German exports are not being substantially damaged by the strong euro\u2014indeed , German costs are so competitive that exporters are confident they can cope with the euro at 1.50 dollars and even higher . This is not true for France-- but the answer is obvious : reduce costs and France too will be able to cope with the strong euro . In the final analysis , cutting costs is all the Europeans can do if they want to preserve the competitiveness of their exports . Th
 e Chinese are not going to change their exchange rate regime , which has strategic as well as economic motivations , just because the Europeans are unhappy with the currency peg . China has given the Europeans a gift . They should be grateful , rather than foolishly insist that China take it back . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/6syndicate_china-s-threat-to-russia.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/6syndicate_china-s-threat-to-russia.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/6syndicate_china-s-threat-to-russia.txt
new file mode 100644
index 0000000..38c905a
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/6syndicate_china-s-threat-to-russia.txt
@@ -0,0 +1,2 @@
+
+China\u2019s Threat to Russia Last week , Russia and China held joint military maneuvers in the presence of both Russian President Vladimir Putin and Chinese President Hu Jintao . But a new strategic alliance between the two countries is not likely , as it is China that poses the greatest strategic threat to Russia , although many in the Kremlin seem blind to this as they rattle sabers at America and the West . Indeed , China officially considers several regions in Russia\u2019s Far East to be only \u201c alienated \u201d from it . China\u2019s territorial claims on Russia are often noted in Chinese grade school geography textbooks , which include a number of Russian Far Eastern regions within China\u2019s borders . This is consistent with the Chinese strategic concept of \u201c vital space , \u201d which includes all spheres of a state\u2019s strategic activities , on land , at sea , under water , in the air , and in space . The dimensions of \u201c vital space \u201d are determined by a country\u2019s economic , sci
 entific , technical , social , and military capabilities \u2013 in essence , its \u201c total power . \u201d According to Chinese theorists , the \u201c vital space \u201d of great powers extends far beyond a state\u2019s borders , whereas the \u201c vital space \u201d of weak countries is limited to strategic boundaries that do not always correspond to the borders of their national territory . Today , China has territorial claims against 11 of its 24 neighbors , including India , Japan , Vietnam , and the Philippines , in addition to Russia . In China\u2019s relations with all of them , the potential use of military force was and remains an important factor . In September 2006 , China\u2019s People\u2019s Liberation Army ( PLA ) conducted an unprecedentedly large 10-day exercise , involving the Shenyang and Beijing military districts , the two most powerful of China\u2019s seven military districts . Shenyang abuts the Russian Armed Forces \u2019 Far Eastern District , and the Beijing district shares a border with Russia
 \u2019s Siberian Military District . During the exercises , Shenyang units advanced 1,000 kilometers into the Beijing district , where they engaged in joint war games . To military observers , the Beijing/Shenyang exercises seemed to be practice for a possible offensive operation against Russia , because exercises on such a scale are undertaken only at the final stage of a multi-year program to train troops to enact specific strategic and operational plans . The geography of the exercises , and the offensive nature of the tasks undertaken , leaves little doubt that Russia was cast in the role of \u201c potential adversary . \u201d Such a show of force is an ancient , traditional Chinese political technique . Paradoxically , these exercises were undertaken during a period when bilateral political and economic ties appeared on the surface to be at their highest point . Russia has an important place in Chinese geopolitical calculations , as a supplier of both modern weaponry and energy resource
 s needed to continue its modernization . Therefore , the Chinese are doing everything possible to strengthen their economic and political position in Russia , and to draw Russia into their sphere of influence . And China is succeeding , most importantly by consistently reinforcing Putin\u2019s anti-American and anti-Western agenda . While Beijing/Shenyang exercise should have indicated to Russian leaders that China\u2019s intentions toward Russia may not always be benign , Russia\u2019s political and military leadership seem not to sense any threat ; on the contrary , they continue to sell the Chinese advanced weapons . Russia\u2019s current diplomatic tilt , indeed , is clearly against its own long-term national security interests . China will never be interested in Russia\u2019s economic and political modernization , for it prefers Russia to remain a source of mineral and energy resources and a vast \u201c strategic rear \u201d in its looming challenge with the United States . Likewise , the Shanghai 
 Cooperation Organization ( SCO ) , which just concluded its annual meeting , is in China\u2019s eyes a tool of regional policy that helps strengthen China\u2019s influence and control over Central Asia\u2019s natural resources at the expense of Russia . No nation threatens China\u2019s land borders . China can solve its domestic problems , such as separatism , by itself . China is militarily self-sufficient and needs military cooperation under the SCO framework only in order to free its hands if any conflict should arise that affects its interests . In fact , conflict between Russia and China is possible precisely in Central Asia , given the clear differences in the two countries \u2019 economic and political interests in that region . Aside from control of the region\u2019s energy supplies , water has become a potential source of conflict , given China\u2019s serious shortages . Yet , while the Chinese clearly understand these contingencies and are preparing themselves to deal with them diplomatically 
 and militarily , the Kremlin remains myopically obsessed with the phantom threat of America . Thus , as the Kremlin dreams of re-establishing its domination over what Russians refer to as the \u201c near abroad \u201d ( Ukraine , Belarus , the Baltic countries , and the other post-Soviet states ) , China is increasingly looking at Russia as its own \u201c near abroad . \u201d Will the Kremlin finally wake up to this ? Thirty-six years ago , Richard Nixon and Mao Zedong turned world politics upside down , as both America and China realized that it was the Soviet Union , and not each other , that posed the greater threat . Vladimir Putin needs his own \u201c Nixon moment . \u201d Alienating the West is a foolish strategy when the greatest long term threat to Russia comes from the East . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/7syndicate_europe-s-vital-french-connection-by-kemal-dervi.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/7syndicate_europe-s-vital-french-connection-by-kemal-dervi.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/7syndicate_europe-s-vital-french-connection-by-kemal-dervi.txt
new file mode 100644
index 0000000..8f7007f
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/7syndicate_europe-s-vital-french-connection-by-kemal-dervi.txt
@@ -0,0 +1,2 @@
+
+Europe\u2019s Vital French Connection PARIS \u2013 In the debates raging over the future of the European Union and the eurozone , Germany always takes center stage . It has the largest economy , accounting for 28 % of eurozone GDP and 25 % of the eurozone\u2019s population . It is running a current-account surplus that is now larger than China\u2019s \u2013 indeed , the largest in the world in absolute value . And , while weighted majorities can overrule it on some issues , everyone acknowledges that little can be done in the eurozone unless Germany agrees . But the emphasis on Germany , though justified , should not lead to an underestimation of France\u2019s critical role . France not only accounts for roughly 22 % of eurozone GDP and 20 % of its population \u2013 behind only Germany \u2013 but also has the healthiest demography in the eurozone , whereas the German population is projected to decline over the next decade . At the same time , France\u2019s critical role reflects more than its size . Indeed , 
 in terms of influencing outcomes in Europe , France is as important as Germany , for three reasons . First , France is an indispensable link between southern and northern Europe at a time of growing economic and financial division between creditors and debtors ( a fissure that has begun to assume a cultural dimension ) . An active France can play a bridging role , leveraging its strong relationship with Germany ( a friendship that is a pillar of the EU ) and its proximity and cultural affinities to the Mediterranean . France is \u201c southern \u201d in its current-account deficit , but \u201c northern \u201d in its borrowing costs ( slightly above Germany\u2019s ) , owing partly to inflows of capital fleeing the south , as well as to modest but positive economic growth . Moreover , there is no perceived \u201c re-denomination \u201d risk affecting French assets , given markets \u2019 confidence that France will retain the euro . So , while France faces huge economic challenges , its northern and southern 
 features permit it to play a proactive role in the European project\u2019s survival . French President Fran�ois Hollande has already given a rather successful preview of this role , meeting German Chancellor Angela Merkel in Berlin on his first day in office , and , a month later , participating in a high-profile meeting with the Italian and Spanish prime ministers in Rome . Indeed , he took the lead in adding a \u201c growth pact \u201d to the \u201c stability pact \u201d that had been negotiated under Merkel\u2019s leadership . Second , France , under its new center-left government , must demonstrate that the \u201c European model \u201d of a market economy coupled with strong social solidarity can be reformed and strengthened , rather than abandoned \u2013 not just in Europe\u2019s more pragmatic north , but also in its more ideological south . French Socialists will not renounce their traditional commitments ; nor should they . But they now have the opportunity to contribute to the European model\u2019s renewa
 l . Under Hollande , France\u2019s Socialists favor achieving that renewal through a process of social dialogue that convinces rather than imposes , that focuses both on revenue measures and on boosting government efficiency , and that may adopt some of northern Europe\u2019s more successful \u201c flexicurity \u201d policies , which combine greater labor-market flexibility with strong social protection . The reforms should also introduce much greater individual choice , permitting solutions to retirement , education , health , and lifestyle issues that can be more easily tailored to citizens \u2019 specific circumstances and needs . The government of Hollande and Prime Minister Jean-Marc Ayrault has strong majorities at all executive and legislative levels , giving it a rare opportunity to act . If it can renew the European model at home , it will be able to project that success much more widely , particularly in southern Europe , in turn reinforcing confidence and belief in the EU , particularly
  among the young generation . The French center-left must lead in conceiving a vision for Europe in which solidarity and equity reinforce long-term economic strength . Finally , along with the United Kingdom among European countries , France retains more of a global role than Germany has yet acquired . While the United Nations Security Council is outdated in its structure , it still plays a crucial legitimizing role ; and France , not Germany , is a member . In many other international organizations as well , France punches above its weight . Similarly , while France exports much less than Germany outside the EU , many large French enterprises rival Germany\u2019s in global reach and technical know-how . And French is still a global language . In other words , France not only is a link between Europe\u2019s north and south , but also contributes substantially to linking Europe to the rest of the world . Europe needs a renewed vision and effective policies to realize that vision . France
 s Socialist-Green government can play a critical unifying role as Europeans confront their biggest challenge in decades . Its success will be highly consequential \u2013 not least for the political debate that will inform the outcome of Germany\u2019s elections in 2013. 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/8syndicate_exchange-rate-disorder.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/8syndicate_exchange-rate-disorder.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/8syndicate_exchange-rate-disorder.txt
new file mode 100644
index 0000000..3cd4316
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/8syndicate_exchange-rate-disorder.txt
@@ -0,0 +1,2 @@
+
+Exchange Rate Disorder NEW YORK \u2013 Two troubling features of the ongoing economic recovery are the depressed nature of world trade and the early revival of international global payment imbalances . Estimates by the International Monetary Fund and the United Nations indicate that the volume of international trade in 2010 will still be 7 % to 8 % below its 2008 peak , while many or most countries , including industrial nations , are seeking to boost their current accounts . Indeed , if we believe the IMF\u2019s projections , the world economy\u2019s accumulated current-account surpluses would increase by almost $1 trillion between 2009 and 2012 ! This is , of course , impossible , as surpluses and deficits must be in balance for the world economy as a whole . It simply reflects the recessionary ( or deflationary ) force of weak global demand hanging over the world economy . Under these conditions , export-led growth by major economies is a threat to the world economy . This is true for Chi
 na , Germany ( as French Finance Minister Christine Lagarde has consistently reminded her neighbor ) , Japan , and the United States . Countries running surpluses must adopt expansionary policies and appreciate their currencies . More broadly , to the extent that major emerging-market countries will continue to lead the global recovery , they should reduce their current-account surpluses or even generate deficits to help , through increased imports , spread the benefits of their growth worldwide . But , while that implies that emerging-market currencies must strengthen , disorderly appreciations would do more harm than good . To use an American saying , it might mean throwing out the baby ( economic growth ) with the bathwater ( exchange rate appreciation ) . Consider China , which accounts for the largest share by far of world trade among emerging economies . Real appreciation of the renminbi is necessary for a balanced world economic recovery . But disorderly appreciation may seri
 ously affect China\u2019s economic growth by disrupting its export industries , which would generate major adverse effects on all of East Asia . China needs a major internal restructuring from exports and investments , its two engines of growth in past decades , to personal and government consumption ( education , health , and social protection in the latter case ) . But this restructuring will tend to reduce , not increase , import demand , as exports and investment are much more import-intensive than consumption . Moreover , a sharp appreciation of the renminbi could risk domestic deflation and a financial crisis . Chinese authorities certainly seem to have that interpretation of the roots of Japan\u2019s malaise in mind as they seek to avoid rapid revaluation . The only desirable scenario , therefore , is a Chinese economy that transmits its stimulus to the rest of the world mainly through rising imports generated by rapid economic growth ( i. e. , the income effect on import demand ) 
 , rather than by exchange-rate appreciation ( the substitution effect ) . This requires maintaining rapid growth while undertaking a major but necessarily gradual domestic restructuring , for which a smooth appreciation is much better suited . Now consider other major emerging markets . Here currency appreciation is already taking place , pushed by massive capital inflows since the second quarter of 2009 , and in some cases it can already be said to be excessive ( for example , in Brazil ) . These countries can , of course , resist upward pressure on their currencies by accumulating foreign-exchange reserves , like they did before the global financial crisis . The result is , of course , paradoxical : private funds that flow into these countries are recycled into US Treasury securities via investment of accumulated reserves . Why should emerging-market countries \u2019 central banks undertake this peculiar financial intermediation , which represents a major cost , as the yield of priva
 te funds is higher than that of reserves ? The implication here is that relying on free movement of capital to achieve exchange-rate appreciation and current-account deficits may generate a myriad of problems , including slower economic growth and the threat of asset bubbles and financial crises of their own . So , a more orderly way to induce current-account deficits without risking disruption of emerging economies \u2019 growth should be considered . One solution ( already advocated by some , including me , and adopted to some extent by a few countries ) is broader use of capital-account regulations . Surprisingly , however , this issue has been entirely absent from current global debates on financial reform . Fortunately , the IMF opened the door to discussion of this issue in a recent staff position paper . Equally important , a desirable global scenario is possibly one in which most developing countries run current-account deficits . But this requires major reforms in the global f
 inancial system to reduce the vulnerabilities that such deficits generated in the past , and that were reflected in major financial crises in the developing world . These past crises gave rise to a form of \u201c self-insurance \u201d among developing countries through reserve accumulation . This helped many of them weather the recent storm , but it also contributed to global payments imbalances . Recent IMF reforms are just a step in the direction of trying to create better financial instruments to help these countries . It is essential , in particular , to create reliable large-scale financing for developing countries during crises , through a mix of counter-cyclical issuance of SDRs and emergency financing without onerous conditions . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/9syndicate_france-and-germany-must-act-in-iraq.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/9syndicate_france-and-germany-must-act-in-iraq.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/9syndicate_france-and-germany-must-act-in-iraq.txt
new file mode 100644
index 0000000..ce55ce0
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/9syndicate_france-and-germany-must-act-in-iraq.txt
@@ -0,0 +1,2 @@
+
+France and Germany Must Act in Iraq Next week , President Bush , President Chirac , and Chancellor Schroeder will meet on the cliffs of Normandy to mark the 60th anniversary of the D-Day invasions that led to the liberation of Europe . They should also seize the moment to liberate themselves from the bitterness that has divided the Atlantic Alliance over the Iraq War , argue Pierre Lellouche and Christoph Bertram . When the United States launched the war against Saddam Hussein 's Iraq , France and Germany rightly warned that the invasion could well end up worsening instability in the Middle East and increase the threat of radical Islamic terror . But now that America and its coalition have proven themselves to be incapable of bringing stability to Iraq on their own , the French and German governments can no longer rest and smugly say , " I told you so , " as the situation deteriorates even further . Both governments must now become seriously and fully engaged in what must now be a u
 nited Western effort . Of course , it would be best for everyone - France and Germany included - if the current US-led coalition were to succeed and Iraq could turn into a pillar of Middle East stability and modernization . Yet , however desirable this outcome , it is no longer likely ( if it ever was ) . Internal stability and economic recovery continue to be elusive , with the specter of civil war hovering ever closer . A divided Iraq at war with itself would be a disaster for the region , for America 's international credibility and authority , and for transatlantic relations . In an already fragile region , a major Arab country like Iraq would most likely prompt interventions by its nervous neighbors - of which there is no shortage - if it turns into a failed state . Instead of the rule of law , there would be the law of the mafia , of terrorist groups , and of tribal militias . Many of them are already in place . Instead of moving toward modernization and development , the regi
 on would be confirmed as a cockpit for conflict , a threat to itself and to the world beyond . Were the US forced to withdraw in failure , Islamic terrorist groups would claim a historic victory , promising more bloodshed for the West . Moreover , America might also once again retreat bitterly into isolationism , blaming her failure on the UN and on disloyal European allies . So the stakes are high in Iraq , for Europeans no less than for Americans . That is why it is time that even those who have been most skeptical about US policy in Iraq start to do something about it . Both Jacques Chirac and Gerhard Schroeder have repeatedly pronounced that what happens in Iraq is a strategic challenge to their countries . Yet their actions in response to this insight have been modest . The recent turmoil in Iraq has , on the contrary , strengthened those in both capitals who argue that there is now even less reason to get involved on the ground . At the very least , France and Germany are dema
 nding that the Bush administration openly admit to its failure and the need for help . Yet not only is such a demand na�ve , with the Bush team battling for re-election , it is not at all certain that most of America 's European allies would heed a US call for solidarity if one were issued . Confronting the threat that an unsettled Iraq poses to Europe and to the transatlantic relationship implies stopping these tactical games . The passing of authority in Iraq from the US-led coalition to an Iraqi government and the need for a much stronger UN role now offers the chance and imposes the need for everyone to stop posturing and get serious . For their part , EU governments should provide the UN - on whose involvement in Iraq they have long insisted - with the protective force it needs to prepare nation-wide elections in Iraq later in the year , and ask others to join . A UN staff protected by US marines simply cannot do that job . With the consent of both the US and Iraq 's new leade
 rship , EU governments should also offer to organize an international conference uniting all those with a stake in Iraqi stability , including Iraq 's neighbors , to work out a strategy and commit resources in order to save Iraq from slipping into protracted turmoil . If Europe is to act , France and Germany must lead . Precisely because they opposed the war , they are now the only countries that can restore Europe 's unity of action in promoting stability . France professes to bearing a special responsibility for international order , but so does Germany ; only a few weeks ago , Chancellor Schroeder based his claim to a permanent German seat in the UN Security Council on Germany 's readiness to shoulder such responsibilities . Of all the crisis regions in the world today , instability in Iraq constitutes the greatest challenge to international order . But France and Germany can also block European responsibility for international order and for solidarity with a US facing possible d
 efeat in Iraq . The decision is theirs . It is time for both to realize that the consequences of inaction will be no less severe for the region than for the Atlantic relationship . 
\ No newline at end of file


[11/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/151ted_doris_kearns_goodwin_on_learning_from_past_presidents.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/151ted_doris_kearns_goodwin_on_learning_from_past_presidents.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/151ted_doris_kearns_goodwin_on_learning_from_past_presidents.txt
new file mode 100644
index 0000000..4f8c24d
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/151ted_doris_kearns_goodwin_on_learning_from_past_presidents.txt
@@ -0,0 +1,2 @@
+
+So , indeed , I have spent my life looking into the lives of presidents who are no longer alive . Waking up with Abraham Lincoln in the morning , thinking of Franklin Roosevelt when I went to bed at night . But when I try and think about what I 've learned about the meaning in life , my mind keeps wandering back to a seminar that I took when I was a graduate student at Harvard with the great psychologist Erik Erikson . He taught us that the richest and fullest lives attempt to achieve an inner balance between three realms : work , love and play . And that to pursue one realm to the disregard of the other , is to open oneself to ultimate sadness in older age . Whereas to pursue all three with equal dedication , is to make possible a life filled , not only with achievement , but with serenity . So since I tell stories , let me look back on the lives of two of the presidents I 've studied to illustrate this point -- Abraham Lincoln and Lyndon Johnson . As for that first sphere of work 
 , I think what Abraham Lincoln 's life suggests is that fierce ambition is a good thing . He had a huge ambition . But it was n't simply for office or power or celebrity or fame -- what it was for was to accomplish something worthy enough in life so that he could make the world a little better place for his having lived in it . Even as a child , it seemed , Lincoln dreamed heroic dreams . He somehow had to escape that hard-scrabble farm from which he was born . No schooling was possible for him , except a few weeks here , a few weeks there . But he read books in every spare moment he could find . It was said when he got a copy of the King James Bible or " Aesop 's Fables , " he was so excited he could n't sleep . He could n't eat . The great poet Emily Dickinson once said , " There is no frigate like a book to take us lands away . " How true for Lincoln . Though he never would travel to Europe , he went with Shakespeare 's kings to merry England , he went with Lord Byron 's poetry t
 o Spain and Portugal . Literature allowed him to transcend his surroundings . But there were so many losses in his early life that he was haunted by death . His mother died when he was only nine years old . His only sister , Sarah , in childbirth a few years later . And his first love , Ann Rutledge , at the age of 22. Moreover , when his mother lay dying she did not hold out for him the hope that they would meet in an afterworld . She simply said to him , " Abraham , I 'm going away from you now , and I shall never return . " As a result he became obsessed with the thought that when we die our life is swept away , dust to dust . But only as he grew older did he develop a certain consolation from an ancient Greek notion -- but followed by other cultures as well -- that if you could accomplish something worthy in your life , you could live on in the memory of others . Your honor and your reputation would outlive your earthly existence . And that worthy ambition became his lodestar . 
 It carried him through the one significant depression that he suffered when he was in his early 30s . Three things had combined to lay him low . He had broken his engagement with Mary Todd , not certain he was ready to marry her , but knowing how devastating it was to her that he did that . His one intimate friend , Joshua Speed , was leaving Illinois to go back to Kentucky because Speed 's father had died . And his political career in the state legislature was on a downward slide . He was so depressed that friends worried he was suicidal . They took all knives and razors and scissors from his room . And his great friend Speed went to his side and said , " Lincoln , you must rally or you will die . " He said that , " I would just as soon die right now , but I 've not yet done anything to make any human being remember that I have lived . " So fuelled by that ambition , he returned to the state legislature . He eventually won a seat in Congress . He then ran twice for the Senate , los
 t twice . " Everyone is broken by life , " Ernest Hemingway once said , " but some people are stronger in the broken places . " So then he surprised the nation with an upset victory for the presidency over three far more experienced , far more educated , far more celebrated rivals . And then when he won the general election , he stunned the nation even more by appointing each of these three rivals into his Cabinet . It was an unprecedented act at the time because everybody thought , " He 'll look like a figurehead compared to these people . " They said , " Why are you doing this , Lincoln ? " And he said , " Look , these are the strongest and most able men in the country . The country is in peril . I need them by my side . " But perhaps my old friend , Lyndon Johnson might have put it in less noble fashion , " Better to have your enemies inside the tent pissing out , than outside the tent pissing in . " ( Laughter ) But it soon became clear that Abraham Lincoln would emerge as the u
 ndisputed captain of this unruly team . For each of them soon came to understand that he possessed an unparalleled array of emotional strengths and political skills that proved far more important than the thinness of his external r�sum� . For one thing , he possessed an uncanny ability to empathize with and to think about other peoples ' point of view . He repaired injured feelings that might have escalated into permanent hostility . He shared credit with ease , assumed responsibility for the failure of his subordinates , constantly acknowledged his errors and learned from his mistakes . These are the qualities we should be looking for in our candidates in 2008. ( Applause ) He refused to be provoked by petty grievances . He never submitted to jealousy or brooded over perceived slights . And he expressed his unshakeable convictions in everyday language , in metaphors , in stories . And with a beauty of language , almost as if the Shakespeare and the poetry he had so loved as a chi
 ld had worked their way into his very soul . In 1863 , when the Emancipation Proclamation was signed , he brought his old friend , Joshua Speed , back to the White House . And remembered that conversation of decades before , when he was so sad . And he , pointing to the Proclamation said , " I believe in this measure my fondest hopes will be realized . " But as he was about to put his signature on the Proclamation his own hand was numb and shaking because he had shaken a thousand hands that morning at a New Year 's reception . So he put the pen down . He said , " If ever my soul were in an act , it is in this act . But if I sign with a shaking hand , posterity will say , 'He hesitated . ' " So he waited until he could take up the pen and sign with a bold and clear hand . But even in his wildest dreams , Lincoln could never have imagined how far his reputation would reach . I was so thrilled to find an interview with the great Russian writer , Leo Tolstoy , in a New York newspaper in
  the early 1900s . And in it , Tolstoy told of a trip that he 'd recently made to a very remote area of the Caucasus , where there were only wild barbarians , who had never left this part of Russia . Knowing that Tolstoy was in their midst , they asked him to tell stories of the great men of history . So he said , " I told them about Napoleon and Alexander the Great and Frederick the Great and Julius Caesar , and they loved it . But before I finished , the chief of the barbarians stood up and said , " But wait , you have n't told us about the greatest ruler of them all . We want to hear about that man who spoke with a voice of thunder , who laughed like the sunrise , who came from that place called America , which is so far from here , that if a young man should travel there , he would be an old man when he arrived . Tell us of that man . Tell us of Abraham Lincoln . ' " He was stunned . He told them everything he could about Lincoln . And then in the interview he said , " What made
  Lincoln so great ? Not as great a general as Napoleon , not as great a statesman as Frederick the Great . " But his greatness consisted , and historians would roundly agree , in the integrity of his character and the moral fiber of his being . So in the end that powerful ambition that had carried Lincoln through his bleak childhood had been realized . That ambition that had allowed him to laboriously educate himself by himself , to go through that string of political failures and the darkest days of the war . His story would be told . So as for that second sphere , not of work , but of love -- encompassing family , friends and colleagues -- it , too , takes work and commitment . The Lyndon Johnson that I saw in the last years of his life , when I helped him on his memoirs , was a man who had spent so many years in the pursuit of work , power and individual success , that he had absolutely no psychic or emotional resources left to get him through the days once the Presidency was gon
 e . My relationship with him began on a rather curious level . I was selected as a White House Fellow when I was 24 years old . We had a big dance at the White House . President Johnson did dance with me that night . Not that peculiar -- there were only three women out of the 16 White House Fellows . But he did whisper in my ear that he wanted me to work directly for him in the White House . But it was not to be that simple . For in the months leading up to my selection , like many young people , I 'd been active in the anti-Vietnam War movement , and had written an article against Lyndon Johnson , which unfortunately came out in The New Republic two days after the dance in the White House . ( Laugher ) And the theme of the article was how to remove Lyndon Johnson from power . ( Laughter ) So I was certain he would kick me out of the program . But instead , surprisingly , he said , " Oh , bring her down here for a year , and if I ca n't win her over , no one can . " So I did end up 
 working for him in the White House . Eventually accompanied him to his ranch to help him on those memoirs , never fully understanding why he 'd chosen me to spend so many hours with . I like to believe it was because I was a good listener . He was a great story teller . Fabulous , colorful , anecdotal stories . There was a problem with these stories , however , which I later discovered , which is that half of them were n't true . But they were great , nonetheless . ( Laughter ) So I think that part of his attraction for me was that I loved listening to his tall tales . But I also worried that part of it was that I was then a young woman . And he had had somewhat of a minor league womanizing reputation . So I constantly chatted to him about boyfriends , even when I did n't have any at all . Everything was working perfectly , until one day he said he wanted to discuss our relationship . Sounded very ominous when he took me nearby to the lake , conveniently called Lake Lyndon Baines Jo
 hnson . And there was wine and cheese and a red-checked tablecloth -- all the romantic trappings . And he started out , " Doris , more than any other woman I have ever known ... " And my heart sank . And then he said , " You remind me of my mother . " ( Laughter ) It was pretty embarrassing , given what was going on in my mind . But I must say , the older I 've gotten , the more I realize what an incredible privilege it was to have spent so many hours with this aging lion of a man . A victor in a thousand contests , three great civil rights laws , Medicare , aid to education . And yet , roundly defeated in the end by the war in Vietnam . And because he was so sad and so vulnerable , he opened up to me in ways he never would have had I know him at the height of his power -- sharing his fears , his sorrows and his worries . And I 'd like to believe that the privilege fired within me the drive to understand the inner person behind the public figure , that I 've tried to bring to each o
 f my books since then . But it also brought home to me the lessons which Erik Erikson had tried to instill in all of us , about the importance of finding balance in life . For on the surface , Lyndon Johnson should have had everything in the world to feel good about in those last years , in the sense that he had been elected to the Presidency . He had all the money he needed to pursue any leisure activity he wanted . He owned a spacious ranch in the countryside , a penthouse in the city . Sailboats , speedboats . He had servants to answer any whim , and he had a family who loved him deeply . And yet , years of concentration solely on work and individual success meant that in his retirement he could find no solace in family , in recreation , in sports or in hobbies . It was almost as if the hole in his heart was so large that even the love of a family , without work , could not fill it . As his spirits sagged , his body deteriorated until , I believe , he slowly brought about his own
  death . In those last years , he said he was so sad watching the American people look toward a new President and forgetting him . He spoke with immense sadness in his voice , saying maybe he should have spent more time with his children , and their children in turn . But it was too late . Despite all that power , all that wealth , he was alone when he finally died -- his ultimate terror realized . So as for that third sphere of play , which he never had learned to enjoy , I 've learned over the years that even this sphere requires a commitment of time and energy . Enough so that a hobby , a sport , a love of music , or art , or literature , or any form of recreation , can provide true pleasure , relaxation and replenishment . So deep , for instance , was Abraham Lincoln 's love of Shakespeare , that he made time to spend more than a hundred nights in the theatre , even during those dark days of the war . He said , when the lights went down and a Shakespeare play came on , for a few
  precious hours he could imagine himself back in Prince Hal 's time . But an even more important form of relaxation for him , that Lyndon Johnson never could enjoy , was a love of , somehow , humor . And feeling out what hilarious parts of life can produce as side light to the sadness . He once said that he laughed so he did not cry . That a good story for him was better than a drop of whiskey . His storytelling powers had first been recognized when he was on the circuit in Illinois . The lawyers and the judges would travel from one county courthouse to the other , and when anyone was knowing Lincoln was in town , they would come from miles around to listen to him tell stories . He would stand with his back against a fire and entertain the crowd for hours with his winding tales . And all these stories became part of his memory bank , so he could call on them whenever he needed to . And they 're not quite what you might expect from our marble monument . One of his favorite stories , 
 for example , had to do with the revolutionary war hero , Ethan Allen . And as Lincoln told the story , Mr. Allen went to Britain after the war . And the British people were still upset about losing the revolution , so they decided to embarrass him a little bit by putting a huge picture of General Washington in the only outhouse , where he 'd have to encounter it . They figured he 'd be upset about the indignity of George Washington being in an outhouse . But he came out of the outhouse not upset at all . And so they said , " Well , did you see George Washington in there ? " " Oh , yes , " he said , " perfectly appropriate place for him . " " What do you mean ? " they said . " Well , " he said , " there 's nothing to make an Englishman shit faster than the sight of General George Washington . " ( Laughter ) ( Applause ) So you can imagine , if you are in the middle of a tense cabinet meeting -- and he had hundreds of these stories -- you would have to relax . So between his nightly 
 treks to the theatre , his story telling , and his extraordinary sense of humor and his love of quoting Shakespeare and poetry , he found that form of play which carried him through his days. In my own life , I shall always be grateful for having found a form of play in my irrational love of baseball . Which allows me from the beginning of spring training to the end of the fall to have something to occupy my mind and heart other than my work . It all began when I was only six years old , and my father taught me that mysterious art of keeping score while listening to baseball games . So that when he went to work in New York during the day , I could record for him the history of that afternoon 's Brooklyn Dodgers game . Now , when you 're only six years old , and your father comes home every single night and listens to you -- as I now realize that I , in excruciating detail , recounted every single play of every inning of the game that had just taken place that afternoon . But he made
  me feel I was telling him a fabulous story . It makes you think there 's something magic about history to keep your father 's attention . In fact , I 'm convinced I learned the narrative art from those nightly sessions with my father . Because at first , I 'd be so excited I would blurt out , " The Dodgers won ! " or , " The Dodgers lost ! " Which took much of the drama of this two-hour-telling away . ( Laughter ) So I finally learned you had to tell a story from beginning to middle to end . I must say , so fervent was my love of the old Brooklyn Dodgers in those days , that I had to confess in my first confession two sins that related to baseball . The first occurred because the Dodgers ' catcher , Roy Campanella , came to my hometown of Rockville Centre , Long Island , just as I was in preparation for my first Holy Communion . And I was so excited -- first person I 'd ever see outside of Ebbets Field . But it so happened he was speaking in a Protestant Church . When you are broug
 ht up as a Catholic , you think that if you ever set foot in a Protestant Church , you 'll be struck dead at the threshold . So I went to my father in tears , " What are we going to do ? " He said , " Do n't worry . He 's speaking in a parish hall . We 're sitting in folding chairs . He 's talking about sportsmanship . It 's not a sin . " But as I left that night , I was certain that somehow I 'd traded the life of my everlasting soul for this one night with Roy Campanella . ( Laughter ) And there were no indulgences around that I could buy . So I had this sin on my soul when I went to my first confession . I told the priest right away . He said , " No problem . It was n't a religious service . " But then , unfortunately , he said , " And what else , my child ? " And then came my second sin . I tried to sandwich it in between talking too much in church , wishing harm to others , being mean to my sisters . And he said , " To whom did you wish harm ? " And I had to say that I wished t
 hat various New York Yankees players would break arms , legs and ankles -- ( Laughter ) -- so that the Brooklyn Dodgers could win their first World Series . He said , " How often do you make these horrible wishes ? " And I had to say , every night when I said my prayers . ( Laughter ) So he said , " Look , I 'll tell you something . I love the Brooklyn Dodgers , as you do , but I promise you some day they will win fairly and squarely . You do not need to wish harm on others to make it happen . " " Oh yes , " I said . But luckily , my first confession -- to a baseball-loving priest ! ( Laughter ) Well , though my father died of a sudden heart attack when I was still in my 20s , before I had gotten married and had my three sons , I have passed his memory -- as well as his love of baseball -- on to my boys . Though when the Dodgers abandoned us to come to L. A. , I lost faith in baseball until I moved to Boston and became an irrational Red Socks fan . And I must say , even now , when I
  sit with my sons with our season tickets , I can sometimes close my eyes against the sun and imagine myself , a young girl once more , in the presence of my father , watching the players of my youth on the grassy fields below . Jackie Robinson , Roy Campanella , Pee Wee Reese , and Duke Snider . I must say there is magic in these moments . When I open my eyes and I see my sons in the place where my father once sat , I feel an invisible loyalty and love linking my sons to the grandfather whose face they never had a chance to see , but whose heart and soul they have come to know through all the stories I have told . Which is why in the end , I shall always be grateful for this curious love of history , allowing me to spend a lifetime looking back into the past . Allowing me to learn from these large figures about the struggle for meaning for life . Allowing me to believe that the private people we have loved and lost in our families , and the public figures we have respected in our h
 istory , just as Abraham Lincoln wanted to believe , really can live on , so long as we pledge to tell and to retell the stories of their lives . Thank you for letting me be that storyteller today . ( Applause ) Thank you . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/152ted_elliot_krane_the_mystery_of_chronic_pain.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/152ted_elliot_krane_the_mystery_of_chronic_pain.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/152ted_elliot_krane_the_mystery_of_chronic_pain.txt
new file mode 100644
index 0000000..48e657e
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/152ted_elliot_krane_the_mystery_of_chronic_pain.txt
@@ -0,0 +1,2 @@
+
+I 'm a pediatrician and an anesthesiologist , so I put children to sleep for a living . ( Laughter ) And I 'm an academic , so I put audiences to sleep for free . ( Laughter ) But what I actually mostly do is a manage the pain management service at the Packard Children 's Hospital up at Stanford in Palo Alto . And it 's from the experience from about 20 or 25 years of doing that that I want to bring to you the message this morning , that pain is a disease . Now most of the time , you think of pain as a symptom of a disease . And that 's true most of the time . It 's the symptom of a tumor or an infection or an inflammation or an operation . But about 10 percent of the time , after the patient has recovered from one of those events , pain persists . It persists for months and oftentimes for years . And when that happens , it is its own disease . And before I tell you about how it is that we think that happens and what we can do about it , I want to show you how it feels for my patien
 ts . So imagine , if you will , that I 'm stroking your arm with this feather , as I 'm stroking my arm right now . Now , I want you to imagine that I 'm stroking it with this . Please keep your seat . ( Laughter ) A very different feeling . Now what does it have to do with chronic pain ? Imagine , if you will , these two ideas together . Imagine what your life would be like if I were to stroke it with this feather , but your brain was telling you that this is what you are feeling -- and that is the experience of my patients with chronic pain . In fact , imagine something even worse . Imagine I were to stroke your child 's arm with this feather , and their brain [ was ] telling them that they were feeling this hot torch . That was the experience of my patient , Chandler , whom you see in the photograph . As you can see , she 's a beautiful , young woman . She was 16 years old last year when I met her , and she aspired to be a professional dancer . And during the course of one of her
  dance rehearsals , she fell on her outstretched arm and sprained her wrist . Now you would probably imagine , as she did , that a wrist sprain is a trivial event in a person 's life . Wrap it in an ACE bandage , take some ibuprofen for a week or two , and that 's the end of the story . But in Chandler 's case , that was the beginning of the story . This is what her arm looked like when she came to my clinic about three months after her sprain . You can see that the arm is discolored , purplish in color . It was cadaverically cold to the touch . The muscles were frozen , paralyzed -- dystonic is how we refer to that . The pain had spread from her wrist to her hands , to her fingertips , from her wrist up to her elbow , almost all the way to her shoulder . But the worst part was , not the spontaneous pain that was there 24 hours a day . The worst part was that she had allodynia , the medical term for the phenomenon that I just illustrated with the feather and with the torch . The lig
 htest touch of her arm -- the touch of a hand , the touch even of a sleeve , of a garment , as she put it on -- caused excruciating , burning pain . How can the nervous system get this so wrong ? How can the nervous system misinterpret an innocent sensation like the touch of a hand and turn it into the malevolent sensation of the touch of the flame . Well you probably imagine that the nervous system in the body is hardwired like your house . In your house , wires run in the wall , from the light switch to a junction box in the ceiling and from the junction box to the light bulb . And when you turn the switch on , the light goes on . And when you turn the switch off , the light goes off . So people imagine the nervous system is just like that . If you hit your thumb with a hammer , these wires in your arm -- that , of course , we call nerves -- transmit the information into the junction box in the spinal cord where new wires , new nerves , take the information up to the brain where y
 ou become consciously aware that your thumb is now hurt . But the situation , of course , in the human body is far more complicated than that . Instead of it being the case that that junction box in the spinal cord is just simple where one nerve connects with the next nerve by releasing these little brown packets of chemical information called neurotransmitters in a linear one-on-one fashion , in fact , what happens is the neurotransmitters spill out in three dimensions -- laterally , vertically , up and down in the spinal cord -- and they start interacting with other adjacent cells . These cells , called glial cells , were once thought to be unimportant structural elements of the spinal cord that did nothing more than hold all the important things together , like the nerves . But it turns out the glial cells have a vital role in the modulation , amplification and , in the case of pain , the distortion of sensory experiences . These glial cells become activated . Their DNA starts to
  synthesize new proteins , which spill out and interact with adjacent nerves . And they start releasing their neurotransmitters . And those neurotransmitters spill out and activate adjacent glial cells , and so on and so forth , until what we have is a positive feedback loop . It 's almost as if somebody came into your home and rewired your walls , so that the next time you turned on the light switch , the toilet flushed three doors down , or your dishwasher went on , or your computer monitor turned off . That 's crazy , but that 's , in fact , what happens with chronic pain . And that 's why pain becomes its own disease . The nervous system has plasticity . It changes , and it morphs in response to stimuli . Well , what do we do about that ? What can we do in a case like Chandler 's ? We treat these patients in a rather crude fashion at this point in time . We treat them with symptom-modifying drugs -- pain-killers -- which are , frankly , not very effective for this kind of pain .
  We take nerves that are noisy and active that should be quiet , and we put them to sleep with local anesthetics . And most importantly , what we do is we use a rigorous , and often uncomfortable , process of physical therapy and occupational therapy to retrain the nerves in the nervous system to respond normally to the activities and sensory experiences that are part of everyday life . And we support all of that with an intensive psychotherapy program to address the despondency , despair and depression that always accompanies severe , chronic pain . It 's successful , as you can see from this video of Chandler , who , two months after we first met her , is now doings a back flip . And I had lunch with her yesterday , because she 's a college student studying dance at Long Beach here . And she 's doing absolutely fantastic . But the future is actually even brighter . The future holds the promise that new drugs will be developed that are not symptom-modifying drugs that simply mask t
 he problem , as we have now , but that will be disease-modifying drugs that will actually go right to the root of the problem and attack those glial cells , or those pernicious proteins that the glial cells elaborate , that spill over and cause this central nervous system wind-up , or plasticity , that so is capable of distorting and amplifying the sensory experience that we call pain . So I have hope that in the future , the prophetic words of George Carlin will be realized , who said , " My philosophy : No pain , no pain . " Thank you very much . ( Applause ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/153ted_eva_zeisel_on_the_playful_search_for_beauty.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/153ted_eva_zeisel_on_the_playful_search_for_beauty.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/153ted_eva_zeisel_on_the_playful_search_for_beauty.txt
new file mode 100644
index 0000000..162c6f9
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/153ted_eva_zeisel_on_the_playful_search_for_beauty.txt
@@ -0,0 +1,2 @@
+
+So I understand that this meeting was planned , and the slogan was From Was to Still . And I am illustrating Still . Which , of course , I am not agreeing with because , although I am 94 , I am not still working . And anybody who asks me , " Are you still doing this or that ? " I do n't answer because I 'm not doing things still , I 'm doing it like I always did . I still have -- or did I use the word still ? I did n't mean that . ( Laughter ) I have my file which is called To Do . I have my plans . I have my clients . I am doing my work like I always did . So this takes care of my age . I want to show you my work so you know what I am doing and why I am here . This was about 1925. All of these things were made during the last 75 years . ( Laughter ) ( Applause ) But , of course , I 'm working since 25 , doing more or less what you see here . This is Castleton China . This was an exhibition at the Museum of Modern Art . This is now for sale at the Metropolitan Museum . This is still
  at the Metropolitan Museum now for sale . This is a portrait of my daughter and myself . ( Applause ) These were just some of the things I 've made . I made hundreds of them for the last 75 years . I call myself a maker of things . I do n't call myself an industrial designer because I 'm other things . Industrial designers want to make novel things . Novelty is a concept of commerce , not an aesthetic concept . The industrial design magazine , I believe , is called " Innovation . " Innovation is not part of the aim of my work . Well , makers of things : they make things more beautiful , more elegant , more comfortable than just the craftsmen do . I have so much to say . I have to think what I am going to say . Well , to describe our profession otherwise , we are actually concerned with the playful search for beauty . That means the playful search for beauty was called the first activity of Man . Sarah Smith , who was a mathematics professor at MIT , wrote , " The playful search for
  beauty was Man 's first activity -- that all useful qualities and all material qualities were developed from the playful search for beauty . " These are tiles . The word , " playful " is a necessary aspect of our work because , actually , one of our problems is that we have to make produce lovely things throughout all of life , and this for me is now 75 years . So how can you , without drying up , make things with the same pleasure , as a gift to others , for so long ? The playful is therefore an important part of our quality as designer . Let me tell you some about my life . As I said , I started to do these things 75 years ago . My first exhibition in the United States was at the Sesquicentennial exhibition in 1926 -- that the Hungarian government sent one of my hand-drawn pieces as part of the exhibit . My work actually took me through many countries , and showed me a great part of the world . This is not that they took me -- the work did n't take me -- I made the things particu
 larly because I wanted to use them to see the world . I was incredibly curious to see the world , and I made all these things , which then finally did take me to see many countries and many cultures . I started as an apprentice to a Hungarian craftsman , and this taught me what the guild system was in Middle Ages . The guild system : that means when I was an apprentice , I had to apprentice myself in order to become a pottery master . In my shop where I studied , or learned , there was a traditional hierarchy of master , journeyman and learned worker , and apprentice , and I worked as the apprentice . The work as an apprentice was very primitive . That means I had to actually learn every aspect of making pottery by hand . We mashed the clay with our feet when it came from the hillside . After that , it had to be kneaded . It had to then go in , kind of , a mangle . And then finally it was prepared for the throwing . And there I really worked as an apprentice . My master took me to s
 et ovens because this was part of oven-making , oven-setting , in the time . And finally , I had received a document that I had accomplished my apprenticeship successfully , that I had behaved morally , and this document was given to me by the Guild of Roof-Coverers , Rail-Diggers , Oven-Setters , Chimney Sweeps and Potters . ( Laughter ) I also got at the time a workbook which explained my rights and my working conditions , and I still have that workbook . First I set up a shop in my own garden , and made pottery which I sold on the marketplace in Budapest . And there I was sitting , and my then-boyfriend -- I did n't mean it was a boyfriend like it is meant today -- but my boyfriend and I sat at the market and sold the pots . My mother thought that this was not very proper , so she sat with us to add propriety to this activity . ( Laughter ) However , after a while there was a new factory being built in Budapest , a pottery factory , a large one . And I visited it with several lad
 ies , and asked all sorts of questions of the director . Then the director asked me , why do you ask all these questions ? I said , I also have a pottery . So he asked me , could he please visit me , and then finally he did , and explained to me that what I did now in my shop was an anachronism , that the industrial revolution had broken out , and that I rather should join the factory . There he made an art department for me where I worked for several months . However , everybody in the factory spent his time at the art department . The director there said there were several women casting and producing my designs now in molds , and this was sold also to America . I remember that it was quite successful . However , the director , the chemist , model maker -- everybody -- concerned himself much more with the art department -- that means , with my work -- than making toilets , so finally they got a letter from the center , from the bank who owned the factory , saying , make toilet-sett
 ing behind the art department , and that was my end . So this gave me the possibility because now I was a journeyman , and journeymen also take their satchel and go to see the world . So as a journeyman , I put an ad into the paper that I had studied , that I was a down-to-earth potter 's journeyman and I was looking for a job as a journeyman . And I got several answers , and I accepted the one which was farthest from home and practically , I thought , halfway to America . And that was in Hamburg . Then I first took this job in Hamburg , at an art pottery where everything was done on the wheel , and so I worked in a shop where there were several potters . And the first day , I was coming to take my place at the turntable -- there were three or four turntables -- and one of them , behind where I was sitting , was a hunchback , a deaf-mute hunchback , who smelled very bad . So I doused him in cologne every day , which he thought was very nice , and therefore he brought bread and butte
 r every day , which I had to eat out of courtesy . The first day I came to work in this shop there was on my wheel a surprise for me . My colleagues had thoughtfully put on the wheel where I was supposed to work a very nicely modeled natural man 's organs . ( Laughter ) After I brushed them off with a hand motion , they were very -- I finally was now accepted , and worked there for some six months . This was my first job . If I go on like this , you will be here till midnight . ( Laughter ) ( Applause ) So I will try speed it up a little ( Laughter ) Moderator : Eva , we have about five minutes . ( Laughter ) Eva Zeisel : Are you sure ? Moderator : Yes , I am sure . EZ : Well , if you are sure , I have to tell you that within five minutes I will talk very fast . And actually , my work took me to many countries because I used my work to fill my curiosity . And among other things , other countries I worked , was in the Soviet Union , where I worked from '32 to '37 -- actually , to '36
  . I was finally there , although I had nothing to do -- I was a foreign expert . I became art director of the china and glass industry , and eventually under Stalin 's purges -- at the beginning of Stalin 's purges , I did n't know that hundreds of thousands of innocent people were arrested . So I was arrested quite early in Stalin 's purges , and spent 16 months in a Russian prison . The accusation was that I had successfully prepared an attentate on Stalin 's life . This was a very dangerous accusation . And if this is the end of my five minutes , I want to tell you that I actually did survive , which was a surprise . But since I survived and I 'm here , and since this is the end of the five minutes , I will -- Moderator : Tell me when your last trip to Russia was . Were n't you there recently ? EZ : Oh , this summer , in fact , the Lomonosov factory was bought by an American company , invited me . They found out that I had worked in '33 at this factory , and they came to my stud
 io in Rockland County , and brought the 15 of their artists to visit me here . And they invited myself to come to the Russian factory last summer , in July , to make some dishes , design some dishes . And since I do n't like to travel alone , they also invited my daughter , son-in-law and granddaughter , so we had a lovely trip to see Russia today , which is not a very pleasant and happy view . Here I am now , if this is the end ? Thank you . ( Applause ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/154ted_ex_moonie_diane_benscoter_how_cults_think.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/154ted_ex_moonie_diane_benscoter_how_cults_think.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/154ted_ex_moonie_diane_benscoter_how_cults_think.txt
new file mode 100644
index 0000000..f8f587c
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/154ted_ex_moonie_diane_benscoter_how_cults_think.txt
@@ -0,0 +1,2 @@
+
+My journey to coming here today started in 1974. That 's me with the funny gloves . I was 17 and going on a peace walk . What I did n't know though , was most of those people , standing there with me , were Moonies . ( Laughter ) And within a week I had come to believe that the second coming of Christ had occurred , that it was Sun Myung Moon , and that I had been specially chosen and prepared by God to be his disciple . Now as cool as that sounds , my family was not that thrilled with this . ( Laughter ) And they tried everything they could to get me out of there . There was an underground railroad of sorts that was going on during those years . Maybe some of you remember it . They were called deprogrammers . And after about five long years my family had me deprogrammed . And I then became a deprogrammer . I started going out on cases . And after about five years of doing [ deprogramming ] I was arrested for kidnapping . Most of the cases I went out on were called involuntary . Wha
 t happened was that the family had to get their loved ones some safe place somehow . And so they took them to some safe place . And we would come in and talk to them , usually for about a week . And so after [ the arrest ] happened I decided it was a good time to turn my back on this work . And about 20 years went by . There was a burning question though that would not leave me . And that was , " How did this happen to me ? " And in fact , what did happen to my brain ? Because something did . And so I decided to write a book , a memoir , about this decade of my life . And toward the end of writing that book there was a documentary that came out . It was on Jonestown . And it had a chilling effect on me . These are the dead in Jonestown . About 900 people died that day . Most of them taking their own lives . Women gave poison to their babies , and watched foam come from their mouths as they died . The top picture is a group of Moonies that have been blessed by their messiah . Their m
 ates were chosen for them . The bottom picture is Hitler youth . This is the leg of a suicide bomber . The thing I had to admit to myself , with great repulsion , was that I get it . I understand how this could happen . I understand how someone 's brain , how someone 's mind can come to the place where it makes sense , in fact it would be wrong , when your brain is working like that , not to try to save the world through genocide . And so what is this ? How does this work ? And how I 've come to view what happened to me is a viral memetic infection . For those of you who are n't familiar with memetics , a meme has been defined as an idea that replicates in the human brian and moves from brain to brain like a virus , much like a virus . The way a virus works is -- it can infect and do the most damage to someone who has a compromised immune system . In 1974 , I was young , I was naive , and I was pretty lost in my world . I was really idealistic . These easy ideas to complex questions
  are very appealing when you are emotionally vulnerable . What happens is that circular logic takes over . " Moon is one with God . God is going to fix all the problems in the world . All I have to do is humbly follow . Because God is going to stop war and hunger -- all these things I wanted to do . All I have to do is humbly follow . Because after all , God is [ working through ] the messiah . He 's going to fix all this . " It becomes impenetrable . And the most dangerous part of this is that is creates " us " and " them , " " right " and " wrong , " " good " and " evil . " And it makes anything possible . Makes anything rationalizable . And the thing is , though , if you looked at my brain during those years in the Moonies -- Neuroscience is expanding exponentially , as Ray Kurzweil said yesterday . Science is expanding . We 're beginning to look inside the brain . And so if you looked at my brain , or any brain that 's infected with a viral memetic infection like this , and comp
 ared it to anyone in this room , or anyone who uses critical thinking on a regular basis , I am convinced it would look very , very different . And that , strange as it may sound , gives me hope . And the reason that gives me hope is that the first thing is to admit that we have a problem . But it 's a human problem . It 's a scientific problem , if you will . It happens in the human brain . There is no evil force out there to get us . And so this is something that , through research and education , I believe that we can solve . And so the first step is to realize that we can do this together , and that there is no " us " and " them . " Thank you very much . ( Applause ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/155ted_hans_rosling_at_state.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/155ted_hans_rosling_at_state.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/155ted_hans_rosling_at_state.txt
new file mode 100644
index 0000000..18b05f1
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/155ted_hans_rosling_at_state.txt
@@ -0,0 +1,2 @@
+
+I 'm going to talk about your mindset . Does your mindset correspond to my dataset ? ( Laughter ) If not , one or the other needs upgrading , is n't it ? When I talk to my students about global issues , and I listen to them in the coffee break , they always talk about " we " and " them . " And when they come back into the lecture room I ask them , " What do you mean with " we " and " them " ? " Oh it 's very easy . It 's the western world and it 's the developing world . " They say . " We learned it in college . " And what is the definition then ? " The definition ? Everyone knows , " they say . But then you know , I press them like this . So one girl said , very cleverly , " It 's very easy . Western world is a long life in a small family . Developing world is a short life in a large family . " And I like that definition because it enabled me to transfer their mindset into the dataset . And here you have the dataset . So you can see that what we have on this axis here is size of fa
 mily . One , two , three , four , five children per woman on this axis . And here , length of life , life expectancy , 30 , 40 , 50. Exactly what the students said was their concept about the world . And really this is about the bedroom . Whether man and woman decide to have small family , and take care of their kids , and how long they will live . It 's about the bathroom and the kitchen . If you have soap , water and food , you know , you can live long . And the students were right . It was n't that the world consisted -- the world consisted here , of one set of countries over here , which had large families and short life . Developing world . And we had one set of countries up there which was the western world . They had small families and long life . And you are going to see here the amazing thing that has happened in the world during my lifetime . And then the developing countries applied soap and water , vaccination . And all the developing world start to apply family planning
  . And partly to USA who help to provide technical advice and investment . And you see all the world moves over to a two child family , and a life with 60 to 70 years . But some countries remain back in this area here . And you can see we still have Afghanistan down here . We have Liberia . We have Congo . So we have countries living there . So the problem I had is that the worldview that my students had correspond to reality in the world the year their teachers were born . ( Laughter ) ( Applause ) And we , in fact , when we have played this over the world . I was at the Global Health Conference here in Washington , last week , and I could see that the wrong concept even active people in the United States had . That they did n't realize the improvement of Mexico there , and China , in relation to United States . Look here when I move them forward . Here we go . They catch up . There 's Mexico . It 's on par with United States in these two social dimensions . There was less than fiv
 e percent of the specialists in global health that was aware of this . This great nation , Mexico , has the problem that arms are coming from north , across the borders . So they had to stop that . Because they have this strange relationship to the United States , you know . But if I would change this axis here , look , and I would instead , here , I would put income per person . Income per person . I can put that here . And we will then see a completely different picture . By the way , I 'm teaching you how to use our website , Gapminder World , while I 'm correcting this because this is a free utility on the net . And when I now finally got it right , I can go back 200 years in history . And I can find United States up there . And I can let the other countries be shown . And now I have income per person on this axis . And the United States only had someone 2,000 dollar , at that time . And the life expectancy was 35 to 40 years , on par with Afghanistan today . And what has happen
 ed in the world , I will show now . This is instead of studying history for one year at university . You can watch me for one minute now and you 'll see the whole thing . ( Laughter ) You can see how the brown bubbles , which is west Europe , and the yellow one , which is United States , they get richer and richer and also start to get healthier and healthier . And this is now 100 years ago where the as the rest of the world remains behind . Here we come . And that was the influenza . That 's why we are so scared about flu is n't it ? It 's still remembered . The fall of life expectancy . And then we come up . And not until independence started . Look here you have China over there , you have India over there , and this is what has happened . You note there , that we have Mexico up there . Mexico is not at all on par with the United States . But they are quite close . And especially it 's interesting to see China and the United States , during 200 years . Because I have my oldest so
 n now working for Google , after Google acquired this software . Because in fact this is child labor . My son and his wife sat in a closet for many years and developed this . And my youngest son , who studied Chinese in Beijing . So they come in with the two perspectives I have . You know ? And my son , youngest son who studied in Beijing , in China , he got a long term perspective . Where as when my oldest son , who work in Google , he should develop by quarter , or by half-year . Or , Google is quite generous , so he can have one or two years to go . But in China they look generation after generation because they remember the very embarrassing period , for 100 years , where they went backwards . And then they would remember the first part of the last century , which was really bad . And we could go by this so-called great leap forward . But this was 1963. Mao Tse-Tung eventually brought health to China . And then he died . And then Deng Xiaoping started this amazing move forward .
  Is n't it strange to see that United States first grew the economy , and then gradually got rich . Where as China could get healthy much early . Because they applied the knowledge of education , nutrition , and then also benefits of penicillin and vaccines , and family planning . And Asia could have social development before they got the economic development . So to me , as a public health professor , it 's not strange that all these countries grow so fast now . Because what you see here , what you see here is the flat world of Thomas Friedman . Is n't it ? It 's not really really flat . But the middle income countries , and this is where I suggest to my students , stop using the concept " developing world . " Because after all , talking about developing world is like having two chapters in the history of United States . The last chapter is about the present , and president Obama . And the other is about the past . Where you cover everything from Washington to Eisenhower . Because 
 Washington to Eisenhower , that is what we find in developing world . We could actually go from Mayflower to Eisenhower , and that would be put together into a developing world . Which is rightly growing its cities in a very amazing way . Which have great entrepreneurs , but also have the collapsing countries , So how could we make a better sense about this ? Well one way of trying is to see whether we could look at income distribution . This is the income distribution of peoples in the world , from one dollar . This is where you have food to eat . These people go to bed hungry . And this is the number of people . This is 10 dollar , whether you have a public or a private health service system . This is where you can provide health service for your family , and school for your children . And this is OECD countries . Green , Latin America , East Europe . This is East Asia . And the light blue there is South Asia . And this is how the world changed . It changed like this . Can you see
  how it 's growing ? And how hundreds of millions and billions is coming out of poverty in Asia ? And it goes over here . And I come now , into projections . But I have to stop at the door of Lehman Brothers there . You know . Because ... ( Laughter ) Because there the projections are not valid any longer . Probably the world will do this . And then it will continue forward like this . But more or less this is what will happen . And we have a world which can not be looked upon as divided . We have the high income countries here , with United States as a leading power . We have the emerging economies in the middle , that provide a lot of the funding for the bailout . And we have the low income countries here . Yeah this is a fact that from where the money come . They have been saving , you know , over the last decade . And here we have the low income countries where entrepreneurs are . And here we have the countries in collapse and war , like Afghanistan , Somalia , parts of Congo , 
 Darfur . We have all this at the same time . That 's why it 's so problematic to describe what has happened in the developing world . Because it 's so different , what has happened there . And that 's why I suggest a slightly different approach of what you would call it . And you have huge difference within countries also . I heard that your departments here were by regions . Here you have Sub-Saharan Africa , South Asia , East Asia , Arab states , East Europe , Latin America , and OECD . And on this axis GDP . And on this , heath , child survival . And it does n't come as a surprise that Africa , south of Sahara , is at the bottom . But when I split it , when I split it into country bubbles , the size of the bubbles here is the population . Then you see Sierra Leone and Mauritius is completely different . There is such a difference within Sub-Saharan Africa . And I can split the others . Here the South Asian , Arab world . Now all you different departments . East Europe , Latin Ame
 rica , and OECD countries . And here were are . We have a continuum in the world . We can not put it into two parts . It is Mayflower down here . It is Washington here , building , building countries . It 's Lincoln here , advancing them . It 's Eisenhower bringing modernity into the countries . And then it 's United States today , up here . And we have countries all this way . Now this is the important thing of understanding how the world has changed . At this point I decided to make a pause . ( Laughter ) And it is my task , on behalf of the rest of the world , to convey a thank to the U. S. taxpayers , for Demographic Health Survey . Many are not aware of -- no this is not a joke . This is very serious . It is due to USA 's continuous sponsoring during 25 years of the very good methodology for measuring child mortality that we have a grasp of what 's happening in the world . ( Applause ) And it is U. S. government at its best , without advocacy , providing facts , that it 's usef
 ul for the society . And providing data free of charge , on the internet , for the world to use . Thank you very much . Quite in the opposite of the World Bank , who compiled data with government money , tax money , and then they sell it to add a little profit , in a very inefficient , Guttenberg way . ( Applause ) But the people doing that at the world bank are among the best in the world . And they are highly skilled professionals . It 's just that we would like to upgrade our international agencies to deal with the world in a modern way , as we do . And when it comes to free data and transparency , United States of America is one of the best . And that does n't come easy from the mouth of a Swedish public health professor . ( Laughter ) And I 'm not paid to come here , no . I would like to show you what happens with the data , what we can show with this data . Look here . This is the world . With income down there , and child mortality . And what has happened in the world ? Since
  1950 , during the last 50 years we have had a fall in child mortality . And this is the DHS that makes it possible to know this . And we had an increase in income . And the blue former developing countries are mixing up with the former industrialized western world . And we have a continuum . But we still have , and that is , of course , Congo , up there . We still have as poor countries as we have had , always , in history . And that 's the bottom billion , where we 've heard , today , about a completely new approach to do it . And how fast has this happened ? Well MDG 4. United States have not been so eager to use MDG 4. But you have been the main sponsor that has enabled us to measure it . Because it 's the only child mortality that we can measure . And we used to say that it should fall four percent per year . Let 's see what Sweden have done . We used to boast about fast social progress . That 's where we were , 1900. 1900 , Sweden was there . Same child mortality as Bangladesh
  had , 1990. Though they had lower income . They started very well . They used the aid well . They vaccinated the kids . They get better water . And they reduced child mortality , with an amazing 4.7 percent per year . They beat Sweden . I run Sweden the same 16 year period . Second round it 's Sweden 1916 , against Egypt 1990. Here we go . Once again USA is part of the reason here . They get safe water . They get food for the poor . And they get malaria eradicated . 5.5 percent . They are faster than millennium development goal . And third chance for Sweden , against Brazil here . And Brazil here has amazing social improvement over the last 16 years . And they go faster than Sweden . This means that the world is converging . The middle income countries , the emerging economy , they are catching up . They are moving to cities , where they will also get better assistance for that . What the Swedish do is protest at this time . They say , " This is not fair . because these countries h
 ad vaccine and antibiotic that was not available for Sweden . We have to do real-time competition . " Okay . I give you Singapore , the year I was born . Singapore had twice the child mortality of Sweden . It 's the most tropical country in the world . A marshland on the equator . And here we go . It took a little time for them to get independent . But then they started to grow their economy . And they made the social investment . They got away malaria . They got a magnificent health system that beat both U. S. and Sweden . We never thought it would happen that they would win over Sweden ! ( Applause ) All these green countries are achieving millennium development goals . These yellow are just about to doing this . These red countries that does n't do it , and the policy has to be improved . Not simplistic extrapolation . We have to really find a way of supporting those countries in a better way . We have to respect the middle income countries on what they are doing . And we have to
  fact-base the whole way we look at the world . This is dollar per person . This is HIV in the countries . The blue is Africa . The size of the bubbles is how many are HIV affected . You see the tragedy in South Africa there . About 20 percent of the adult population are infected . And in spite of them having quite a high income they have a huge number of HIV infected . But you also see that there are African countries down here . There is no such thing as an HIV epidemic in Africa . There 's a number , 5 to 10 countries in Africa that has the same level as Sweden and United States . And there are others who are extremely high . And I will show you that what has happened in one of the best countries , with the most vibrant economy in Africa , and a good governance , is Botswana . They have a very high level . It 's coming down . But now it 's not falling . Because there , with help from PEPFAR it 's working with treatment . And people are not dying . And you can see it 's not that e
 asy , that it is war which caused this . Because here , in Congo , there is war . And here , in Zambia , there is peace . And it 's not the economy . Richer country has a little higher . And if I split Tanzania in its income . The richer 20 percent in Tanzania has more HIV than the poorest one . And it 's really different within country . Look at the provinces of Kenya . They are very different . And this is the situation you see . It 's not deep poverty . It 's the special situation . Probably of concurrent sexual partnership among part of the heterosexual population in some countries , or some parts of countries , in south and eastern Africa . Do n't make it Africa . Do n't make it a race issue . Make it a local issue . And do prevention at each place , in the way it can be done there . So to just end up . There are things of suffering in the one billion poorest , which we do n't know . Those who live beyond the cellphone , those who have yet to see a computer , those who have no 
 electricity at home . This is the disease Konzo , I spent 20 years elucidating in Africa . It 's caused by fast processing of toxic cassava root , in famine situation . It 's similar to the pellagra epidemic in Mississippi , in the '30s . It 's similar to other nutritional diseases . It will never affect a rich person We have seen it here in Mozambique . This is the epidemic in Mozambique . This is an epidemic in northern Tanzania . You never heard about the disease . But it 's much more than ebola that has been affected by this disease . Cause crippling throughout the world . And over the last two years 2,000 people has been crippled in the southern tip of Bandunda region . That used to be the illegal diamond trade , from the UNITA-dominated area in Angola . That has now disappeared . And they are now in great economical problem . And one week ago , for the first time , there were four lines on the Internet . Do n't get confused of the progress of the emerging economies , and the g
 reat capacity of people in the middle income countries , and in peaceful low income countries . There is still mystery in one billion . And we have to have more concept than just developing countries and developing world . We need a new mindset . The world is converging . But , but , but , not the bottom billion . They are still as poor as they 've ever been . It 's not sustainable . And it will not happen around one superpower . But you will remain one of the most important superpower . And the most hopeful superpower , for the time to be . And this institution will have a very crucial role , not for United States , but for the world . So you have a very bad name , State Department , this is not the State Department . It 's the World Department . And we have a high hope in you . Thank you very much . ( Applause ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/156ted_helen_fisher_tells_us_why_we_love_cheat.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/156ted_helen_fisher_tells_us_why_we_love_cheat.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/156ted_helen_fisher_tells_us_why_we_love_cheat.txt
new file mode 100644
index 0000000..dd44e56
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/156ted_helen_fisher_tells_us_why_we_love_cheat.txt
@@ -0,0 +1,2 @@
+
+I 'd like to talk today about the two biggest social trends in the coming century , and perhaps in the next 10,000 years . But I want to start with my work on romantic love , because that 's my most recent work . What I and my colleagues did was to put 32 people , who were madly in love , into a functional MRI brain scanner . 17 who were madly in love and their love was accepted ; and 15 who were madly in love and they had just been dumped . And so I want to tell you about that first , and then go on into where I think love is going . " What 'tis to love ? " Shakespeare said . I think our ancestors -- I think human beings have been wondering about this question since they sat around their campfires or lay and watched the stars a million years ago . I started out by trying to figure out what romantic love was by looking at the last 45 years of research on -- just the psychological research , and as it turns out , there 's a very specific group of things that happen when you fall in l
 ove . The first thing that happens is what I call -- a person begins to take on what I call , " special meaning . " As a truck driver once said to me , he said , " The world had a new center , and that center was Mary Anne . " George Bernard Shaw said it a little differently . He said , " Love consists of overestimating the differences between one woman and another . " And indeed , that 's what we do . ( Laughter ) And then you just focus on this person . You can list what you do n't like about them , but then you sweep that aside and focus on what you do . As Chaucer said , " Love is blind . " In trying to understand romantic love , I decided I would read poetry from all over the world , and I just want to give you one very short poem from eighth-century China , because it 's an almost perfect example of a man who is focused totally on a particular woman . It 's a little bit like when you are madly in love with somebody and you walk into a parking lot . Their car is different from 
 every other car in the parking lot . Their wine glass at dinner is different from every other wine glass at the dinner party . And in this case , a man got hooked on a bamboo sleeping mat . And it goes like this . It 's by a guy called Yuan Chen : " I cannot bear to put away the bamboo sleeping mat . The night I brought you home , I watched you roll it out . " He became hooked on a sleeping mat , probably because of elevated activity of dopamine in his brain , just like with you and me . But anyway , not only does this person take on special meaning , you focus your attention on them . You aggrandize them . But you have intense energy . As one Polynesian said , he said , " I felt like jumping in the sky . " You 're up all night . You 're walking till dawn . You feel intense elation when things are going well , mood swings into horrible despair when things are going poorly . Real dependence on this person . As one businessman in New York said to me , he said , " Anything she liked , 
 I liked . " Simple . Romantic love is very simple . You become extremely sexually possessive . You know , if you 're just sleeping with somebody casually , you do n't really care if they 're sleeping with somebody else . But the moment you fall in love , you become extremely sexually possessive of them . I think that that is a Darwinian -- there 's a Darwinian purpose to this . The whole point of this is to pull two people together strongly enough to begin to rear babies as a team . But the main characteristics of romantic love are craving : an intense craving to be with a particular person , not just sexually , but emotionally . You 'd much rather -- it would be nice to go to bed with them , but you want them to call you on the telephone , to invite you out , et cetera . To tell you that they love you . The other main characteristic is motivation . The motor in your brain begins to crank , and you want this person . And last but not least , it is an obsession . When I put these peo
 ple in the machine , before I put them in the MRI machine , I would ask them all kinds of questions . But my most important question was always the same . It was : " What percentage of the day and night do you think about this person ? " And indeed , they would say , " All day . All night . I can never stop thinking about him or her . " And then , the very last question I would ask them -- I would always have to work myself up to this question , because I am not a psychologist . I do n't work with people in any kind of traumatic situation . And my final question was always the same . I would say , " Would you die for him or her ? " And , indeed , these people would say " Yes ! , " as if I had asked them to pass the salt . I was just staggered by it . So we scanned their brains , looking at a photograph of their sweetheart and looking at a neutral photograph , with a distraction task in between . So we could find -- look at the same brain when it was in that heightened state and when
  it was in a resting state . And we found activity in a lot of brain regions . In fact , one of the most important was a brain region that becomes active when you feel the rush of cocaine . And indeed , that 's exactly what happens . I began to realize that romantic love is not an emotion . In fact , I had always thought it was a series of emotions , from very high to very low . But actually , it 's a drive . It comes from the motor of the mind , the wanting part of the mind , the craving part of the mind . The kind of mind -- part of the mind -- when you 're reaching for that piece of chocolate , when you want to win that promotion at work . The motor of the brain . It 's a drive . And in fact , I think it 's more powerful than the sex drive . You know , if you ask somebody to go to bed with you , and they say , " No thank you , " you certainly do n't kill yourself or slip into a clinical depression . But certainly , around the world , people who are rejected in love will kill for 
 it . People live for love . They kill for love . They die for love . They have songs , poems , novels , sculptures , paintings , myths , legends . In over 175 societies , people have left their evidence of this powerful brain system . I have come to think it 's one of the most powerful brain systems on earth for both great joy and great sorrow . And I 've also come to think that it 's one of three , basically different brain systems that evolved from mating and reproduction . One is the sex drive : the craving for sexual gratification . W. H. Auden called it an " intolerable neural itch , " and indeed , that 's what it is . It keeps bothering you a little bit , like being hungry . The second of these three brain systems is romantic love : that elation , obsession of early love . And the third brain system is attachment : that sense of calm and security you can feel for a long-term partner . And I think that the sex drive evolved to get you out there , looking for a whole range of pa
 rtners . You know , you can feel it when you 're just driving along in your car . It can be focused on nobody . I think romantic love evolved to enable you to focus your mating energy on just one individual at a time , thereby conserving mating time and energy . And I think that attachment , the third brain system , evolved to enable you to tolerate this human being -- ( Laughter ) -- at least long enough to raise a child together as a team . So with that preamble , I want to go into discussing the two most profound social trends . One of the last 10,000 years and the other -- certainly of the last 25 years -- that are going to have an impact on these three different brain systems : lust , romantic love and deep attachment to a partner . The first is women working , moving into the workforce . I 've looked at 150 -- 130 societies through the demographic yearbooks of the United Nations . And everywhere in the world , 129 out of 130 of them , women are not only moving into the job mar
 ket -- sometimes very , very slowly , but they are moving into the job market -- and they are very slowly closing that gap between men and women in terms of economic power , health and education . It 's very slow . For every trend in -- on this planet , there 's a counter-trend . We all know of them , but nevertheless -- the old Arab saying . The Arabs say , " The dogs may bark , but the caravan moves on . " And , indeed , that caravan is moving on . Women are moving back into the job market . And I say back into the job market , because this is not new . For millions of years , on the grasslands of Africa , women commuted to work to gather their vegetables . They came home with 60 to 80 percent of the evening meal . The double income family was the standard . And women were regarded as just as economically , socially and sexually powerful as men . In short , we 're really moving forward to the past . Then , women 's worst invention was the plow . With the beginning of plow agricult
 ure , men 's roles became extremely powerful . Women lost their ancient jobs as collectors , but then with the industrial revolution and the post-industrial revolution they 're moving back into the job market . In short , they are acquiring the status that they had a million years ago , 10,000 years ago , 100,000 years ago . We are seeing now one of the most remarkable traditions in the history of the human animal . And it 's going to have an impact . I generally give a whole lecture on the impact of women on the business community . I 'll only just say a couple of things , and then go on to sex and love . There 's a lot of gender differences ; anybody who thinks men and women are alike simply never had a boy and a girl child . I do n't know why it is that they want to think that men and women are alike . There 's much we have in common , but there 's a whole lot that we are not -- do not have in common . We are -- in the words of Ted Hughes , " I think that we were built to be -- w
 e 're like two feet . We need each other to get ahead . " But we did not evolve to have the same brain . And we 're finding more and more and more gender differences in the brain . I 'll only just use a couple and then move on to sex and love . One of them is women 's verbal ability . Women can talk . Women 's ability to find the right word rapidly , basic articulation goes up in the middle of the menstrual cycle , when estrogen levels peak . But even at menstruation , they 're better than the average man . Women can talk . They 've been doing it for a million years ; words were women 's tools . They held that baby in front of their face , cajoling it , reprimanding it , educating it with words . And , indeed , they 're becoming a very powerful force . Even in places like India and Japan , where women are not moving rapidly into the regular job market , they 're moving into journalism . And I think that the television is like the global campfire . We sit around it and it shapes our 
 minds . Almost always , when I 'm on TV , the producers who call me , who negotiate what we 're going to say , is a woman . In fact , Solzhenitsyn once said , " To have a great writer is to have another government . " Today 54 percent of people who are writers in America are women . It 's one of many , many characteristics that women have that they will bring into the job market . They 've got incredible people skills , negotiating skills . They 're highly imaginative . We now know the brain circuitry of imagination , of long-term planning . They tend to be web thinkers . Because the female parts of the brain are better connected , they tend to collect more pieces of data when they think , put them into more complex patterns , see more options and outcomes . They tend to be contextual , holistic thinkers , what I call web thinkers . Men tend to -- and these are averages -- tend to get rid of what they regard as extraneous , focus on what they do , and move in a more step-by-step thi
 nking pattern . They 're both perfectly good ways of thinking . We need both of them to get ahead . In fact , there 's many more male geniuses in the world . When the -- and there 's also many more male idiots in the world . ( Laughter ) When the male brain works well , it works extremely well . And I -- what I really think that we 're doing is , we 're moving towards a collaborative society , a society in which the talents of both men and women are becoming understood and valued and employed . But in fact , women moving into the job market is having a huge impact on sex and romance and family life . Foremost , women are starting to express their sexuality . I 'm always astonished when people come to me and say , " Why is it that men are so adulterous ? " And I say , " Why do you think more men are adulterous than women ? " " Oh , well -- men are more adulterous ! " And I say , " Who do you think these men are sleeping with ? " And -- basic math ! ( Laughter ) Anyway . In the Wester
 n world , little girls start -- women start sooner at sex , have more partners , express less remorse for the partners that they do , marry later , have fewer children , leave bad marriages in order to get good ones . We are seeing the rise of female sexual expression . And , indeed , once again we 're moving forward to the kind of sexual expression that we probably saw on the grasslands of Africa a million years ago , because this is the kind of sexual expression that we see in hunting and gathering societies today . We 're also returning to an ancient form of marriage equality . They 're now saying that the 21st century is going to be the century of what they call the " symmetrical marriage , " or the " pure marriage , " or the " companionate marriage . " This is a marriage between equals , moving forward to a pattern that is highly compatible with the ancient human spirit . We 're also seeing a rise of romantic love . 91 percent of American women and 86 percent of American men wo
 uld not marry somebody who had every single quality they were looking for in a partner , if they were not in love with that person . People around the world , in a study of 37 societies , want to be in love with the person that they marry . Indeed , arranged marriages are on their way off this braid of human life . I even think that marriages might even become more stable because of the second great world trend . The first one being women moving into the job market , the second one being the aging world population . They 're now saying that in America , that middle age should be regarded as up to age 85. Because in that highest age category of 76 to 85 , only -- as much as 40 percent of people have nothing really wrong with them . So we 're seeing there 's a real extension of middle age . And I looked -- for one of my books , I looked at divorce data in 58 societies . And as it turns out , the older you get , the less likely you are to divorce . So the divorce rate right now is stab
 le in America , and it 's actually beginning to decline . It may decline some more . I would even say that with Viagra , estrogen replacement , hip replacements and the incredibly interesting women -- women have never been as interesting as they are now . Not at any time on this planet have women been so educated , so interesting , so capable . And so I honestly think that if there really was ever a time in human evolution when we have the opportunity to make good marriages , that time is now . However , there 's always kinds of complications in this . In these three brain systems : lust , romantic love and attachment -- do n't always go together . They can go together , by the way . That 's why casual sex is n't so casual . With orgasm you get a spike of dopamine . Dopamine 's associated with romantic love , and you can just fall in love with somebody who you 're just having casual sex with . With orgasm , then you get a real rush of oxytocin and vasopressin -- those are associated
  with attachment . This is why you can feel such a sense of cosmic union with somebody after you 've made love to them . But these three brain systems : lust , romantic love and attachment , are n't always connected to each other . You can feel deep attachment to a long-term partner while you feel intense romantic love for somebody else , while you feel the sex drive for people unrelated to these other partners . In short , we 're capable of loving more than one person at a time . In fact , you can lie in bed at night and swing from deep feelings of attachment for one person to deep feelings of romantic love for somebody else . It 's as if there 's a committee meeting going on in your head as you are trying to decide what to do . So I do n't think , honestly , we 're an animal that was built to be happy ; we are an animal that was built to reproduce . I think the happiness we find , we make . And I think , however , we can make good relationships with each other . So I want to concl
 ude with two things . I want to conclude with a worry . I have a worry -- and with a wonderful story . The worry is about antidepressants . Over 100 million prescriptions of antidepressants are written every year in the United States . And these drugs are going generic . They are seeping around the world . I know one girl who 's been on these antidepressants , serotonin-enhancing -- SSRI , serotonin-enhancing antidepressants -- since she was 13. She 's 23. She 's been on them ever since she was 13. I 've got nothing against people who take them short term , when they 're going through something perfectly horrible . They want to commit suicide or kill somebody else . I would recommend it . But more and more people in the United States are taking them long term . And indeed , what these drugs do is raise levels of serotonin . And by raising levels of serotonin , you suppress the dopamine circuit . Everybody knows that . Dopamine is associated with romantic love . Not only do they supp
 ress the dopamine circuit , but they kill the sex drive . And when you kill the sex drive , you kill orgasm . And when you kill orgasm , you kill that flood of drugs associated with attachment . The things are connected in the brain . And when you tamper with one brain system , you 're going to tamper with another . I 'm just simply saying that a world without love is a deadly place . So now -- ( Applause ) -- thank you . I want to end with a story . And then , just a comment . I 've been studying romantic love and sex and attachment for 30 years . I 'm an identical twin ; I am interested in why we 're all alike . Why you and I are alike , why the Iraqis and the Japanese and the Australian Aborigines and the people of the Amazon River are all alike . And about a year ago , an Internet dating service , Match . com , came to me and asked me if I would design a new dating site for them . I said , " I do n't know anything about personality . You know ? I do n't know . Do you think you '
 ve got the right person ? " They said , " Yes . " It got me thinking about why it is that you fall in love with one person rather than another . That 's my current project ; it will be my next book . There 's all kinds of reasons that you fall in love with one person rather than another . Timing is important . Proximity is important . Mystery is important . You fall in love with somebody who 's somewhat mysterious , in part because mystery elevates dopamine in the brain , probably pushes you over that threshold to fall in love . You fall in love with somebody who fits within what I call your " love map , " an unconscious list of traits that you build in childhood as you grow up . And I also think that you become -- gravitate to certain people , actually , with somewhat complementary brain systems . And that 's what I 'm now contributing to this . But I want to tell you a story about -- to illustrate . I 've been carrying on here about the biology of love . I wanted to show you a lit
 tle bit about the culture of it , too -- the magic of it . It 's a story that was told to me by somebody who had heard it just from one of the -- probably a true story . It was a graduate student at -- I 'm at Rutgers and my two colleagues -- Art Aron is at SUNY Stony Brook . That 's where we put our people in the MRI machine . And this graduate student was madly in love with another graduate student , and she was not in love with him . And they were all at a conference in Beijing . And he knew from our work that if you go and do something very novel with somebody , you can drive up the dopamine in the brain . And perhaps trigger this brain system for romantic love . ( Laughter ) So he decided he 'd put science to work , and he invited this girl to go off on a rickshaw ride with him . And sure enough -- I 've never been in one , but apparently they go all around the buses and the trucks and it 's crazy and it 's noisy and it 's exciting . And he figured that this would drive up the 
 dopamine , and she would fall in love with him . So off they go and she 's squealing and squeezing him and laughing and having a wonderful time . An hour later they get down off of the rickshaw , and she throws her hands up and she says , " Was n't that wonderful ? " And , " Was n't that rickshaw driver handsome ! " ( Laughter ) ( Applause ) There 's magic to love ! But I will end by saying that millions of years ago , we evolved three basic drives : the sex drive , romantic love and attachment to a long-term partner . These circuits are deeply embedded in the human brain . They 're going to survive as long as our species survives on what Shakespeare called " this mortal coil . " Thank you . ( Applause ) 
\ No newline at end of file


[27/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Marx/77MarxMarxK_GothaProg_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Marx/77MarxMarxK_GothaProg_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Marx/77MarxMarxK_GothaProg_EN.txt.txt
new file mode 100644
index 0000000..2ce01d5
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Marx/77MarxMarxK_GothaProg_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Critique of the Gotha Programme -------------------------------------------------------------------------------- I -------------------------------------------------------------------------------- 1. " Labor is the source of wealth and all culture , and since useful labor is possible only in society and through society , the proceeds of labor belong undiminished with equal right to all members of society . " First part of the paragraph : " Labor is the source of all wealth and all culture . " Labor is not the source of all wealth . Nature is just as much the source of use values ( and it is surely of such that material wealth consists ! ) as labor , which itself is only the manifestation of a force of nature , human labor power . the above phrase is to be found in all children 's primers and is correct insofar as it is implied that labor is performed with the appurtenant subjects and instruments . But a socialist program cannot allow such bourgeois phrases to pass over in silence
  the conditions that lone give them meaning . And insofar as man from the beginning behaves toward nature , the primary source of all instruments and subjects of labor , as an owner , treats her as belonging to him , his labor becomes the source of use values , therefore also of wealth . The bourgeois have very good grounds for falsely ascribing supernatural creative power to labor ; since precisely from the fact that labor depends on nature it follows that the man who possesses no other property than his labor power must , in all conditions of society and culture , be the slave of other men who have made themselves the owners of the material conditions of labor . He can only work with their permission , hence live only with their permission . Let us now leave the sentence as it stands , or rather limps . What could one have expected in conclusion ? Obviously this : " Since labor is the source of all wealth , no one in society can appropriate wealth except as the product of labor . 
 Therefore , if he himself does not work , he lives by the labor of others and also acquires his culture at the expense of the labor of others . " Instead of this , by means of the verbal river " and since " , a proposition is added in order to draw a conclusion from this and not from the first one . Second part of the paragraph : " Useful labor is possible only in society and through society . " According to the first proposition , labor was the source of all wealth and all culture ; therefore no society is possible without labor . Now we learn , conversely , that no " useful " labor is possible without society . One could just as well have said that only in society can useless and even socially harmful labor become a branch of gainful occupation , that only in society can one live by being idle , etc. , etc. -- in short , once could just as well have copied the whole of Rousseau . And what is " useful " labor ? Surely only labor which produces the intended useful result . A savage 
 -- and man was a savage after he had ceased to be an ape -- who kills an animal with a stone , who collects fruit , etc. , performs " useful " labor . Thirdly , the conclusion : " Useful labor is possible only in society and through society , the proceeds of labor belong undiminished with equal right to all members of society . " A fine conclusion ! If useful labor is possible only in society and through society , the proceeds of labor belong to society -- and only so much therefrom accrues to the individual worker as is not required to maintain the " condition " of labor , society . In fact , this proposition has at all times been made use of by the champions of the state of society prevailing at any given time . First comes the claims of the government and everything that sticks to it , since it is the social organ for the maintenance of the social order ; then comes the claims of the various kinds of private property , for the various kinds of private property are the foundations
  of society , etc. One sees that such hollow phrases are the foundations of society , etc. One sees that such hollow phrases can be twisted and turned as desired . The first and second parts of the paragraph have some intelligible connection only in the following wording : " Labor becomes the source of wealth and culture only as social labor " , or , what is the same thing , " in and through society " . This proposition is incontestably correct , for although isolated labor ( its material conditions presupposed ) can create use value , it can create neither wealth nor culture . But equally incontestable is this other proposition : " In proportion as labor develops socially , and becomes thereby a source of wealth and culture , poverty and destitution develop among the workers , and wealth and culture among the nonworkers . " This is the law of all history hitherto . What , therefore , had to be done here , instead of setting down general phrases about " labor " and " society " , was
  to prove concretely how in present capitalist society the material , etc. , conditions have at last been created which enable and compel the workers to lift this social curse . In fact , however , the whole paragraph , bungled in style and content , is only there in order to inscribe the Lassallean catchword of the " undiminished proceeds of labor " as a slogan at the top of the party banner . I shall return later to the " proceeds of labor " , " equal right " , etc. , since the same thing recurs in a somewhat different form further on . 2. " In present-day society , the instruments of labor are the monopoly of the capitalist class ; the resulting dependence of the working class is the cause of misery and servitude in all forms . " This sentence , borrowed from the Rules of the International , is incorrect in this " improved " edition . In present-day society , the instruments of labor are the monopoly of the landowners ( the monopoly of property in land is even the basis of the mo
 nopoly of capital ) and the capitalists . In the passage in question , the Rules of the International do not mention either one or the other class of monopolists . They speak of the " monopolizer of the means of labor , that is , the sources of life . " The addition , " sources of life " , makes it sufficiently clear that land is included in the instruments of labor . The correction was introduced because Lassalle , for reasons now generally known , attacked only the capitalist class and not the landowners . In England , the capitalist class is usually not even the owner of the land on which his factory stands . 3. " The emancipation of labor demands the promotion of the instruments of labor to the common property of society and the co-operative regulation of the total labor , with a fair distribution of the proceeds of labor . " Promotion of the instruments of labor to the common property " ought obviously to read their " conversion into the common property " ; but this is only pas
 sing . What are the " proceeds of labor " ? The product of labor , or its value ? And in the latter case , is it the total value of the product , or only that part of the value which labor has newly added to the value of the means of production consumed ? " Proceeds of labor " is a loose notion which Lassalle has put in the place of definite economic conceptions . What is " a fair distribution " ? Do not the bourgeois assert that the present-day distribution is " fair " ? And is it not , in fact , the only " fair " distribution on the basis of the present-day mode of production ? Are economic relations regulated by legal conceptions , or do not , on the contrary , legal relations arise out of economic ones ? Have not also the socialist sectarians the most varied notions about " fair " distribution ? To understand what is implied in this connection by the phrase " fair distribution " , we must take the first paragraph and this one together . The latter presupposes a society wherein t
 he instruments of labor are common property and the total labor is co-operatively regulated , and from the first paragraph we learn that " the proceeds of labor belong undiminished with equal right to all members of society . " " To all members of society " ? To those who do not work as well ? What remains then of the " undiminished " proceeds of labor ? Only to those members of society who work ? What remains then of the " equal right " of all members of society ? But " all members of society " and " equal right " are obviously mere phrases . The kernel consists in this , that in this communist society every worker must receive the " undiminished " Lassallean " proceeds of labor " . Let us take , first of all , the words " proceeds of labor " in the sense of the product of labor ; then the co-operative proceeds of labor are the total social product . From this must now be deducted : First , cover for replacement of the means of production used up . Second , additional portion for e
 xpansion of production . Third , reserve or insurance funds to provide against accidents , dislocations caused by natural calamities , etc. These deductions from the " undiminished " proceeds of labor are an economic necessity , and their magnitude is to be determined according to available means and forces , and partly by computation of probabilities , but they are in no way calculable by equity . There remains the other part of the total product , intended to serve as means of consumption . Before this is divided among the individuals , there has to be deducted again , from it : First , the general costs of administration not belonging to production . This part will , from the outset , be very considerably restricted in comparison with present-day society , and it diminishes in proportion as the new society develops . Second , that which is intended for the common satisfaction of needs , such as schools , health services , etc. From the outset , this part grows considerably in com
 parison with present-day society , and it grows in proportion as the new society develops . Third , funds for those unable to work , etc. , in short , for what is included under so-called official poor relief today . Only now do we come to the " distribution " which the program , under Lassallean influence , alone has in view in its narrow fashion -- namely , to that part of the means of consumption which is divided among the individual producers of the co-operative society . The " undiminished " proceeds of labor have already unnoticeably become converted into the " diminished " proceeds , although what the producer is deprived of in his capacity as a private individual benefits him directly or indirectly in his capacity as a member of society . Just as the phrase of the " undiminished " proceeds of labor has disappeared , so now does the phrase of the " proceeds of labor " disappear altogether . Within the co-operative society based on common ownership of the means of production ,
  the producers do not exchange their products ; just as little does the labor employed on the products appear here as the value of these products , as a material quality possessed by them , since now , in contrast to capitalist society , individual labor no longer exists in an indirect fashion but directly as a component part of total labor . The phrase " proceeds of labor " , objectionable also today on account of its ambiguity , thus loses all meaning . What we have to deal with here is a communist society , not as it has developed on its own foundations , but , on the contrary , just as it emerges from capitalist society ; which is thus in every respect , economically , morally , and intellectually , still stamped with the birthmarks of the old society from whose womb it emerges . Accordingly , the individual producer receives back from society -- after the deductions have been made -- exactly what he gives to it . What he has given to it is his individual quantum of labor . For 
 example , the social working day consists of the sum of the individual hours of work ; the individual labor time of the individual producer is the part of the social working day contributed by him , his share in it . He receives a certificate from society that he has furnished such-and-such an amount of labor ( after deducting his labor for the common funds ) ; and with this certificate , he draws from the social stock of means of consumption as much as the same amount of labor cost . The same amount of labor which he has given to society in one form , he receives back in another . Here , obviously , the same principle prevails as that which regulates the exchange of commodities , as far as this is exchange of equal values . Content and form are changed , because under the altered circumstances no one can give anything except his labor , and because , on the other hand , nothing can pass to the ownership of individuals , except individual means of consumption . But as far as the dis
 tribution of the latter among the individual producers is concerned , the same principle prevails as in the exchange of commodity equivalents : a given amount of labor in one form is exchanged for an equal amount of labor in another form . Hence , equal right here is still in principle -- bourgeois right , although principle and practice are no longer at loggerheads , while the exchange of equivalents in commodity exchange exists only on the average and not in the individual case . In spite of this advance , this equal right is still constantly stigmatized by a bourgeois limitation . The right of the producers is proportional to the labor they supply ; the equality consists in the fact that measurement is made with an equal standard , labor . But one man is superior to another physically , or mentally , and supplies more labor in the same time , or can labor for a longer time ; and labor , to serve as a measure , must be defined by its duration or intensity , otherwise it ceases to 
 be a standard of measurement . This equal right is an unequal right for unequal labor . It recognizes no class differences , because everyone is only a worker like everyone else ; but it tacitly recognizes unequal individual endowment , and thus productive capacity , as a natural privilege . It is , therefore , a right of inequality , in its content , like every right . Right , by its very nature , can consist only in the application of an equal standard ; but unequal individuals ( and they would not be different individuals if they were not unequal ) are measurable only by an equal standard insofar as they are brought under an equal point of view , are taken from one definite side only -- for instance , in the present case , are regarded only as workers and nothing more is seen in them , everything else being ignored . Further , one worker is married , another is not ; one has more children than another , and so on and so forth . Thus , with an equal performance of labor , and henc
 e an equal in the social consumption fund , one will in fact receive more than another , one will be richer than another , and so on . To avoid all these defects , right , instead of being equal , would have to be unequal . But these defects are inevitable in the first phase of communist society as it is when it has just emerged after prolonged birth pangs from capitalist society . Right can never be higher than the economic structure of society and its cultural development conditioned thereby . In a higher phase of communist society , after the enslaving subordination of the individual to the division of labor , and therewith also the antithesis between mental and physical labor , has vanished ; after labor has become not only a means of life but life 's prime want ; after the productive forces have also increased with the all-around development of the individual , and all the springs of co-operative wealth flow more abundantly -- only then then can the narrow horizon of bourgeois 
 right be crossed in its entirety and society inscribe on its banners : From each according to his ability , to each according to his needs ! I have dealt more at length with the " undiminished " proceeds of labor , on the one hand , and with " equal right " and " fair distribution " , on the other , in order to show what a crime it is to attempt , on the one hand , to force on our Party again , as dogmas , ideas which in a certain period had some meaning but have now become obsolete verbal rubbish , while again perverting , on the other , the realistic outlook , which it cost so much effort to instill into the Party but which has now taken root in it , by means of ideological nonsense about right and other trash so common among the democrats and French socialists . Quite apart from the analysis so far given , it was in general a mistake to make a fuss about so-called distribution and put the principal stress on it . Any distribution whatever of the means of consumption is only a con
 sequence of the distribution of the conditions of production themselves . The latter distribution , however , is a feature of the mode of production itself . The capitalist mode of production , for example , rests on the fact that the material conditions of production are in the hands of nonworkers in the form of property in capital and land , while the masses are only owners of the personal condition of production , of labor power . If the elements of production are so distributed , then the present-day distribution of the means of consumption results automatically . If the material conditions of production are the co-operative property of the workers themselves , then there likewise results a distribution of the means of consumption different from the present one . Vulgar socialism ( and from it in turn a section of the democrats ) has taken over from the bourgeois economists the consideration and treatment of distribution as independent of the mode of production and hence the pre
 sentation of socialism as turning principally on distribution . After the real relation has long been made clear , why retrogress again ? 4. " The emancipation of labor must be the work of the working class , relative to which all other classes are only one reactionary mass . " The first strophe is taken from the introductory words of the Rules of the International , but " improved " . There it is said : " The emancipation of the working class must be the act of the workers themselves " ; here , on the contrary , the " working class " has to emancipate -- what ? " Labor . " Let him understand who can . In compensation , the antistrophe , on the other hand , is a Lassallean quotation of the first water : " relative to which " ( the working class ) " all other classes are only one reactionary mass . " In the Communist Manifesto it is said : " Of all the classes that stand face-to-face with the bourgeoisie today , the proletariat alone is a really revolutionary class . The other classe
 s decay and finally disappear in the face of modern industry ; the proletariat is its special and essential product . " The bourgeoisie is here conceived as a revolutionary class -- as the bearer of large-scale industry -- relative to the feudal lords and the lower middle class , who desire to maintain all social positions that are the creation of obsolete modes of production . thus , they do not form together with the bourgeoisie " only one reactionary mass " . On the other hand , the proletariat is revolutionary relative to the bourgeoisie because , having itself grown up on the basis of large-scale industry , it strives to strip off from production the capitalist character that the bourgeoisie seeks to perpetuate . But the Manifesto adds that the " lower middle class " is becoming revolutionary " in view of [ its ] impending transfer to the proletariat " . From this point of view , therefore , it is again nonsense to say that it , together with the bourgeoisie , and with the feud
 al lords into the bargain , " form only one reactionary mass " relative to the working class . Has one proclaimed to the artisan , small manufacturers , etc. , and peasants during the last elections : Relative to us , you , together with the bourgeoisie and feudal lords , form one reactionary mass ? Lassalle knew the Communist Manifesto by heart , as his faithful followers know the gospels written by him . If , therefore , he has falsified it so grossly , this has occurred only to put a good color on his alliance with absolutist and feudal opponents against the bourgeoisie . In the above paragraph , moreover , his oracular saying is dragged in by main force without any connection with the botched quotation from the Rules of the International . Thus , it is simply an impertinence , and indeed not at all displeasing to Herr Bismarck , one of those cheap pieces of insolence in which the Marat of Berlin deals . [ Marat of Berlin a reference to Hasselmann , cheif editor of the Neuer Soci
 al-Demokrat ] 5. " The working class strives for its emancipation first of all within the framework of the present-day national states , conscious that the necessary result of its efforts , which are common to the workers of all civilized countries , will be the international brotherhood of peoples . " Lassalle , in opposition to the Communist Manifesto and to all earlier socialism , conceived the workers ' movement from the narrowest national standpoint . He is being followed in this -- and that after the work of the International ! It is altogether self-evident that , to be able to fight at all , the working class must organize itself at home as a class and that its own country is the immediate arena of its struggle -- insofar as its class struggle is national , not in substance , but , as the Communist Manifesto says , " in form " . But the " framework of the present-day national state " , for instance , the German Empire , is itself , in its turn , economically " within the fram
 ework " of the world market , politically " within the framework " of the system of states . Every businessman knows that German trade is at the same time foreign trade , and the greatness of Herr Bismarck consists , to be sure , precisely in his pursuing a kind of international policy . And to what does the German Workers ' party reduce its internationalism ? To the consciousness that the result of its efforts will be " the international brotherhood of peoples " -- a phrase borrowed from the bourgeois League of Peace and Freedom , which is intended to pass as equivalent to the international brotherhood of working classes in the joint struggle against the ruling classes and their governments . Not a word , therefore , about the international functions of the German working class ! And it is thus that it is to challenge its own bourgeoisie -- which is already linked up in brotherhood against it with the bourgeois of all other countries -- and Herr Bismarck 's international policy of 
 conspiracy . In fact , the internationalism of the program stands even infinitely below that of the Free Trade party . The latter also asserts that the result of its efforts will be " the international brotherhood of peoples " . But it also does something to make trade international and by no means contents itself with the consciousness that all people are carrying on trade at home . The international activity of the working classes does not in any way depend on the existence of the International Working Men 's Association . This was only the first attempt to create a central organ for the activity ; an attempt which was a lasting success on account of the impulse which it gave but which was no longer realizable in its historical form after the fall of the Paris Commune . Bismarck 's Norddeutsche was absolutely right when it announced , to the satisfaction of its master , that the German Workers ' party had sworn off internationalism in the new program . II " Starting from these bas
 ic principles , the German workers ' party strives by all legal means for the free state\u2014and\u2014socialist society : that abolition of the wage system together with the iron law of wages -- and\u2014exploitation in every form ; the elimination of all social and political inequality . " I shall return to the " free " state later . So , in future , the German Workers ' party has got to believe in Lassalle 's " iron law of wages " ! That this may not be lost , the nonsense is perpetrated of speaking of the " abolition of the wage system " ( it should read : system of wage labor ) , " together with the iron law of wages " . If I abolish wage labor , then naturally I abolish its laws also , whether they are of " iron " or sponge . But Lassalle 's attack on wage labor turns almost solely on this so-called law . In order , therefore , to prove that Lassalle 's sect has conquered , the " wage system " must be abolished " together with the iron law of wages " and not without it . It is well kno
 wn that nothing of the " iron law of wages " is Lassalle 's except the word " iron " borrowed from Goethe 's " great , eternal iron laws " . [ 1 ] The word " iron " is a label by which the true believers recognize one another . But if I take the law with Lassalle 's stamp on it , and consequently in his sense , then I must also take it with his substantiation for it . And what is that ? As Lange already showed , shortly after Lassalle 's death , it is the Malthusian theory of population ( preached by Lange himself ) . But if this theory is correct , then again I cannot abolish the law even if I abolish wage labor a hundred times over , because the law then governs not only the system of wage labor but every social system . Basing themselves directly on this , the economists have been proving for 50 years and more that socialism cannot abolish poverty , which has its basis in nature , but can only make it general , distribute it simultaneously over the whole surface of society ! But 
 all this is not the main thing . Quite apart from the false Lassallean formulation of the law , the truly outrageous retrogression consists in the following : Since Lassalle 's death , there has asserted itself in our party the scientific understanding that wages are not what they appear to be -- namely , the value , or price , of labor\u2014but only a masked form for the value , or price , of labor power . Thereby , the whole bourgeois conception of wages hitherto , as well as all the criticism hitherto directed against this conception , was thrown overboard once and for all . It was made clear that the wage worker has permission to work for his own subsistence\u2014that is , to live , only insofar as he works for a certain time gratis for the capitalist ( and hence also for the latter 's co-consumers of surplus value ) ; that the whole capitalist system of production turns on the increase of this gratis labor by extending the working day , or by developing the productivity\u2014that is , i
 ncreasing the intensity or labor power , etc. ; that , consequently , the system of wage labor is a system of slavery , and indeed of a slavery which becomes more severe in proportion as the social productive forces of labor develop , whether the worker receives better or worse payment . And after this understanding has gained more and more ground in our party , some return to Lassalle 's dogma although they must have known that Lassalle did not know what wages were , but , following in the wake of the bourgeois economists , took the appearance for the essence of the matter . It is as if , among slaves who have at last got behind the secret of slavery and broken out in rebellion , a slave still in thrall to obsolete notions were to inscribe on the program of the rebellion : Slavery must be abolished because the feeding of slaves in the system of slavery cannot exceed a certain low maximum ! Does not the mere fact that the representatives of our party were capable of perpetrating suc
 h a monstrous attack on the understanding that has spread among the mass of our party prove , by itself , with what criminal levity and with what lack of conscience they set to work in drawing up this compromise program ! Instead of the indefinite concluding phrase of the paragraph , " the elimination of all social and political inequality " , it ought to have been said that with the abolition of class distinctions all social and political inequality arising from them would disappear of itself . III -------------------------------------------------------------------------------- " The German Workers ' party , in order to pave the way to the solution of the social question , demands the establishment of producers ' co-operative societies with state aid under the democratic control of the toiling people . The producers ' co-operative societies are to be called into being for industry and agriculture on such a scale that the socialist organization of the total labor will arise from the
 m . " After the Lassallean " iron law of wages " , the physic of the prophet . The way to it is " paved " in worthy fashion . In place of the existing class struggle appears a newspaper scribbler 's phrase : " the social question " , to the " solution " of which one " paves the way " . Instead of arising from the revolutionary process of transformation of society , the " socialist organization of the total labor " " arises " from the " state aid " that the state gives to the producers ' co-operative societies and which the state , not the workers , " calls into being " . It is worthy of Lassalle 's imagination that with state loans one can build a new society just as well as a new railway ! From the remnants of a sense of shame , " state aid " has been put -- under the democratic control of the " toiling people " . In the first place , the majority of the " toiling people " in Germany consists of peasants , not proletarians . Second , " democratic " means in German " Volksherrschaft
 lich " [ by the rule of the people ] . But what does " control by the rule of the people of the toiling people " mean ? And particularly in the case of a toiling people which , through these demands that it puts to the state , expresses its full consciousness that it neither rules nor is ripe for ruling ! It would be superfluous to deal here with the criticism of the recipe prescribed by Buchez in the reign of Louis Philippe , in opposition to the French socialists and accepted by the reactionary workers , of the Atelier . The chief offense does not lie in having inscribed this specific nostrum in the program , but in taking , in general , a retrograde step from the standpoint of a class movement to that of a sectarian movement . That the workers desire to establish the conditions for co-operative production on a social scale , and first of all on a national scale , in their own country , only means that they are working to revolutionize the present conditions of production , and it
  has nothing in common with the foundation of co-operative societies with state aid . But as far as the present co-operative societies are concerned , they are of value only insofar as they are the independent creations of the workers and not prot�g�s either of the governments or of the bourgeois . IV -------------------------------------------------------------------------------- I come now to the democratic section . A. " The free basis of the state . " First of all , according to II , the German Workers ' party strives for " the free state " . Free state \u2014 what is this ? It is by no means the aim of the workers , who have got rid of the narrow mentality of humble subjects , to set the state free . In the German Empire , the " state " is almost as " free " as in Russia . Freedom consists in converting the state from an organ superimposed upon society into one completely subordinate to it ; and today , too , the forms of state are more free or less free to the extent that they 
 restrict the " freedom of the state " . The German Workers ' party \u2014 at least if it adopts the program \u2014 shows that its socialist ideas are not even skin-deep ; in that , instead of treating existing society ( and this holds good for any future one ) as the basis of the existing state ( or of the future state in the case of future society ) , it treats the state rather as an independent entity that possesses its own intellectual , ethical , and libertarian bases . And what of the riotous misuse which the program makes of the words " present-day state " , " present-day society " , and of the still more riotous misconception it creates in regard to the state to which it addresses its demands ? " Present-day society " is capitalist society , which exists in all civilized countries , more or less free from medieval admixture , more or less modified by the particular historical development of each country , more or less developed . On the other hand , the " present-day state " change
 s with a country 's frontier . It is different in the Prusso-German Empire from what it is in Switzerland , and different in England from what it is in the United States . The " present-day state " is therefore a fiction . Nevertheless , the different states of the different civilized countries , in spite or their motley diversity of form , all have this in common : that they are based on modern bourgeois society , only one more or less capitalistically developed . They have , therefore , also certain essential characteristics in common . In this sense , it is possible to speak of the " present-day state " in contrast with the future , in which its present root , bourgeois society , will have died off . The question then arises : What transformation will the state undergo in communist society ? In other words , what social functions will remain in existence there that are analogous to present state functions ? This question can only be answered scientifically , and one does not get 
 a flea-hop nearer to the problem by a thousand-fold combination of the word 'people ' with the word 'state ' . Between capitalist and communist society there lies the period of the revolutionary transformation of the one into the other . Corresponding to this is also a political transition period in which the state can be nothing but the revolutionary dictatorship of the proletariat . Now the program does not deal with this nor with the future state of communist society . Its political demands contain nothing beyond the old democratic litany familiar to all : universal suffrage , direct legislation , popular rights , a people 's militia , etc. They are a mere echo of the bourgeois People 's party , of the League of Peace and Freedom . They are all demands which , insofar as they are not exaggerated in fantastic presentation , have already been realized . Only the state to which they belong does not lie within the borders of the German Empire , but in Switzerland , the United States 
 , etc. This sort of " state of the future " is a present-day state , although existing outside the " framework " of the German Empire . But one thing has been forgotten . Since the German Workers ' party expressly declares that it acts within " the present-day national state " , hence within its own state , the Prusso-German Empire \u2014 its demands would indeed be otherwise largely meaningless , since one only demands what one has not got \u2014 it should not have forgotten the chief thing , namely , that all those pretty little gewgaws rest on the recognition of the so-called sovereignty of the people and hence are appropriate only in a democratic republic . Since one has not the courage \u2014 and wisely so , for the circumstances demand caution \u2014 to demand the democratic republic , as the French workers ' programs under Louis Philippe and under Louis Napoleon did , one should not have resorted , either , to the subterfuge , neither " honest " [ 1 ] nor decent , of demanding things whi
 ch have meaning only in a democratic republic from a state which is nothing but a police-guarded military despotism , embellished with parliamentary forms , alloyed with a feudal admixture , already influenced by the bourgeoisie , and bureaucratically carpentered , and then to assure this state into the bargain that one imagines one will be able to force such things upon it " by legal means " . Even vulgar democracy , which sees the millennium in the democratic republic , and has no suspicion that it is precisely in this last form of state of bourgeois society that the class struggle has to be fought out to a conclusion \u2014 even it towers mountains above this kind of democratism , which keeps within the limits of what is permitted by the police and not permitted by logic . That , in fact , by the word " state " is meant the government machine , or the state insofar as it forms a special organism separated from society through division of labor , is shown by the words " the German Wo
 rkers ' party demands as the economic basis of the state : a single progressive income tax " , etc. Taxes are the economic basis of the government machinery and of nothing else . In the state of the future , existing in Switzerland , this demand has been pretty well fulfilled . Income tax presupposes various sources of income of the various social classes , and hence capitalist society . It is , therefore , nothing remarkable that the Liverpool financial reformers \u2014 bourgeois headed by Gladstone 's brother \u2014 are putting forward the same demand as the program . B. " The German Workers ' party demands as the intellectual and ethical basis of the state : " 1. Universal and equal elementary education by the state . Universal compulsory school attendance . Free instruction . " " Equal elementary education " ? What idea lies behind these words ? Is it believed that in present-day society ( and it is only with this one has to deal ) education can be equal for all classes ? Or is it dem
 anded that the upper classes also shall be compulsorily reduced to the modicum of education \u2014 the elementary school \u2014 that alone is compatible with the economic conditions not only of the wage-workers but of the peasants as well ? " Universal compulsory school attendance . Free instruction . " The former exists even in Germany , the second in Switzerland and in the United States in the case of elementary schools . If in some states of the latter country higher education institutions are also " free " , that only means in fact defraying the cost of education of the upper classes from the general tax receipts . Incidentally , the same holds good for " free administration of justice " demanded under A , 5. The administration of criminal justice is to be had free everywhere ; that of civil justice is concerned almost exclusively with conflicts over property and hence affects almost exclusively the possessing classes . Are they to carry on their litigation at the expense of the natio
 nal coffers ? This paragraph on the schools should at least have demanded technical schools ( theoretical and practical ) in combination with the elementary school . " Elementary education by the state " is altogether objectionable . Defining by a general law the expenditures on the elementary schools , the qualifications of the teaching staff , the branches of instruction , etc. , and , as is done in the United States , supervising the fulfillment of these legal specifications by state inspectors , is a very different thing from appointing the state as the educator of the people ! Government and church should rather be equally excluded from any influence on the school . Particularly , indeed , in the Prusso-German Empire ( and one should not take refuge in the rotten subterfuge that one is speaking of a " state of the future " ; we have seen how matters stand in this respect ) the state has need , on the contrary , of a very stern education by the people . But the whole program , f
 or all its democratic clang , is tainted through and through by the Lassallean sect 's servile belief in the state , or , what is no better , by a democratic belief in miracles ; or rather it is a compromise between these two kinds of belief in miracles , both equally remote from socialism . " Freedom of science " says paragraph of the Prussian Constitution . Why , then , here ? . " Freedom of conscience " ! If one desired , at this time of the Kulturkampf to remind liberalism of its old catchwords , it surely could have been done only in the following form : Everyone should be able to attend his religious as well as his bodily needs without the police sticking their noses in . But the Workers ' party ought , at any rate in this connection , to have expressed its awareness of the fact that bourgeois " freedom of conscience " is nothing but the toleration of all possible kinds of religious freedom of conscience , and that for its part it endeavours rather to liberate the conscience f
 rom the witchery of religion . But one chooses not to transgress the " bourgeois " level . I have now come to the end , for the appendix that now follows in the program does not constitute a characteristic component part of it . Hence , I can be very brief . ------------- Appendix -------------------------------------------------------------------------------- " 2. Normal working day . " In no other country has the workers ' party limited itself to such an indefinite demand , but has always fixed the length of the working day that it considers normal under the given circumstances . " 3. Restriction of female labor and prohibition of child labor . " The standardization of the working day must include the restriction of female labor , insofar as it relates to the duration , intermissions , etc. , of the working day ; otherwise , it could only mean the exclusion of female labor from branches of industry that are especially unhealthy for the female body , or are objectionable morally fo
 r the female sex . If that is what was meant , it should have been said so . " Prohibition of child labor . " Here it was absolutely essential to state the age limit . A general prohibition of child labor is incompatible with the existence of large-scale industry and hence an empty , pious wish . Its realization -- if it were possible -- would be reactionary , since , with a strict regulation of the working time according to the different age groups and other safety measures for the protection of children , an early combination of productive labor with education is one of the most potent means for the transformation of present-day society . " 4. State supervision of factory , workshop , and domestic industry . " In consideration of the Prusso-German state , it should definitely have been demanded that the inspectors are to be removable only by a court of law ; that any worker can have them prosecuted for neglect of duty ; that they must belong to the medical profession . " 5. Regula
 tion of prison labor . " A petty demand in a general workers ' program . In any case , it should have been clearly stated that there is no intention from fear of competition to allow ordinary criminals to be treated like beasts , and especially that there is no desire to deprive them of their sole means of betterment , productive labor . This was surely the least one might have expected from socialists . " 6. An effective liability law . " It should have been stated what is meant by an " effective " liability law . Be it noted , incidentally , that , in speaking of the normal working day , the part of factory legislation that deals with health regulations and safety measures , etc. , has been overlooked . The liability law comes into operation only when these regulations are infringed . In short , this appendix also is distinguished by slovenly editing . Dixi et salvavi animam meam . [ I have spoken and saved my soul . ] 
\ No newline at end of file


[24/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/101TediRoslingH_Poverty_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/101TediRoslingH_Poverty_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/101TediRoslingH_Poverty_EN.txt.txt
new file mode 100644
index 0000000..d40be93
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/101TediRoslingH_Poverty_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff I told you three things last year . I told you that the statistics of the world have not been made properly available . Because of that , we still have the old mindset of developing and industrialized countries , which is wrong . And that animated graphics can make a difference . Things are changing . And today , on the United Nations Statistic Division Home Page , it says , by first of May , full access to the databases . ( Applause ) And if I could share the image with you on the screen . So three things had happened . U. N. opened their statistic databases , and we have a new version of the software up working as a beta on the net , so you do n't have to download it any longer . And let me repeat what you saw last year . The bubbles are the countries . Here you have the fertility rate -- the number of children per woman -- and there you have the length of life in years . This is 1950 -- those were the industrialized countries , those were developing countries . At that time t
 here was a " we " and " them . " There was a huge difference in the world . But then it changed , and it went on quite well . And this is what happens . You can see how China is the red , big bubble ; the blue there is India . And they go over all this ... I 'm going to try to be a little more serious this year in showing you how things really changed . And it 's Africa which stands out as the problem down here , does n't it ? Large families still , and the HIV epidemic brought down the countries like this . This is more or less what we saw last year , and this is how it will go on into the future . And I will talk on , is this possible ? Because you see now , I presented statistics that do n't exist . Because this is where we are . Will it be possible that this will happen ? I cover my lifetime here , you know ? I expect to live 100 years . And this is where we are today . Now could we look here at instead the economic situation in the world ? And I would like to show that against 
 child survival . We 'll swap the axis : here you have child mortality -- that is , survival -- four kids dying there , 200 dying there . And this is GDP per capita on this axis . And this was 2007. And if I go back in time , I 've added some historical statistics -- here we go , here we go , here we go -- not so much statistics 100 years ago . Some countries still had statistics . We are looking down in the archive , and where we are down into 1820 , there is only Austria and Sweden that can produce numbers . ( Laughter ) But they were down here , they had 1,000 dollars per person per year . And they lost one-fifth of their kids before their first birthday . So this is what happens in the world , if we play the entire world . How they got slowly richer and richer , and they add statistics . Is n't it beautiful when they get statistics ? You see the importance of that ? And here , children do n't live longer . The last century , 1870 , was bad for the kids in Europe , because most of
  this statistics is Europe . It was only by the turn of the century that more than 90 percent of the children survived their first year . This is India coming up , with the first data from India . And this is the United States moving away here , earning more money . And we will soon see China coming up in the very far end corner here . And it moves up with Mao Tse-Tung getting health , not getting so rich . There he died , then Deng Xiaoping brings money , it moves this way over here . And the bubbles keep moving up there , and this is what the world looks like today . ( Applause ) Let us have a look at the United States . We have a function here -- I can tell the world , " Stay where you are . " And I take the United States -- we still want to see the background -- I put them up like this , and now we go backwards . And we can see that the United States goes to the right of the mainstream . They are on the money side all the time . And down in 1915 , the United States was a neighbo
 r of India -- present , contemporary India . And that means United States was richer , but lost more kids than India is doing today , proportionally . And look here -- compare to the Philippines of today . The Philippines of today has almost the same economy as the United States during the First World War . But we have to bring United States forward quite a while to find the same health of the United States as we have in the Philippines . About 1957 here , the health of the United States is the same as the Philippines . And this is the drama of this world which many call globalized , is that Asia , Arabic countries , Latin America , are much more ahead in being healthy , educated , having human resources than they are economically . There 's a discrepancy in what 's happening today in the emerging economies . There now , social benefits , social progress , are going ahead of economical progress . And 1957 -- the United States had the same economy as Chile has today . And how long do
  we have to bring United States to get the same health as Chile has today ? I think we have to go , there -- we have 2001 , or 2002 -- the United States has the same health than Chile . Chile 's catching up ! Within some years Chile may have better child survival than the United States . This is really a change , that you have this lag of more or less 30 , 40 years ' difference on the health . And behind the health is the educational level . And there 's a lot of infrastructure things , and general human resources are there . Now we can take away this -- and I would like to show you the rate of speed , the rate of change , how fast they have gone . And we go back to 1920 , and I want to look at Japan . And I want to look at Sweden and the United States . And I 'm going to stage a race here between this sort of yellowish Ford here and the red Toyota down there , and the brownish Volvo . ( Laughter ) And here we go , here we go . The Toyota has a very bad start down here , you can see
  , and the United States Ford is going off-road there . And the Volvo is doing quite fine . This is the war . The Toyota got off track , and now the Toyota is coming on the healthier side of Sweden -- can you see that ? And they are taking over Sweden , and they are now healthier than Sweden . That 's the part where I sold the Volvo and bought the Toyota . ( Laughter ) And now we can see that the rate of change was enormous in Japan . They really caught up . And this changes gradually . We have to look over generations to understand it . And let me show you my own sort of family history -- we made these graphs here . And this is the same thing , money down there , and health , you know ? And this is my family . This is Sweden , 1830 , when my great-great-grandma was born . Sweden was like Sierra Leone today . And this is when great-grandma was born , 1863. And Sweden was like Mozambique . And this is when my grandma was born , 1891. She took care of me as a child , so I 'm not talki
 ng about statistic now -- now it 's oral history in my family . That 's when I believe statistics , when it 's grandma-verified statistics . ( Laughter ) I think it 's the best way of verifying historical statistics . Sweden was like Ghana . It 's interesting to see the enormous diversity within sub-Saharan Africa . I told you last year , I 'll tell you again , my mother was born in Egypt , and I -- who am I ? I 'm the Mexican in the family . And my daughter , she was born in Chile , and the grand-daughter was born in Singapore , now the healthiest country on this Earth . It bypassed Sweden about two to three years ago , with better child survival . But they 're very small , you know . They 're so close to the hospital we can never beat them out in these forests . ( Laughter ) But homage to Singapore . Singapore are the best ones , now . Now this looks also like a very good story . But it 's not really that easy , that it 's all a good story . Because I have to show you one of the o
 ther facility . We can also make the color here represent the variable -- and what am I choosing here ? Carbon-dioxide emission , metric ton per capita . This is 1962 , and United States was emitting 16 tons per person . And China was emitting 0.6 , and India was emitting 0.32 tons per capita . And what happens when we moved on ? Well , you see the nice story of getting richer and getting healthier -- everyone did it at the cost of emission of carbon dioxide . There is no one who has done it so far . And we do n't have all the updated data any longer , because this is really hot data today . And there we are , 2001. And in the discussion I attended with global leaders , you know , many say now , the problem is the emerging economies , they are getting out too much carbon dioxide . The Minister of the Environment of India said , " Well , you were the one who caused the problem . " The OECD countries -- the high-income countries -- they were the ones who caused the climate change . " 
 But we forgive you , because you did n't know it . But from now on , we count per capita . From now on we count per capita . And everyone is responsible for the per capita emission . " This really shows you , we have not seen good economic and health progress anywhere in the world without destroying the climate . And this is really what has to be changed . I 've been criticized for showing you a too positive image of the world , but I do n't think it 's like this . The world is quite a messy place . This we can call Dollar Street . Everyone lives on this street here . What they earn here -- what number they live on -- is how much they earn per day . This family earns about one dollar per day . We drive up the street here , we find a family here which earns about two to three dollars a day . And we drive away here -- we find the first garden in the street , and they earn 10 to 50 dollars a day . And how do they live ? If we look at the bed here , we can see that they sleep on a rug o
 n the floor . This is what poverty line is -- 80 percent of the family income is just to cover the energy needs , the food for the day . This is two to five dollars , you have a bed . And here it 's a much nicer bedroom , you can see . I lectured on this for Ikea , and they wanted to see the sofa immediately here . ( Laughter ) And this is the sofa , how it will emerge from there . And the interesting thing , when you go around here in the photo panorama , you see the family still sitting on the floor there , although there is a sofa . If you watch in the kitchen , you can see that the great difference for women does not come between one to 10 dollar . It comes beyond here , when you really can get good working conditions in the family . And if you really want to see the difference , you look at the toilet over here . This can change , this can change . These are all pictures and images from Africa , and it can become much better . We can get out of poverty . My own research has not
  been in IT or anything like this . I spent 20 years in interviews with African farmers who were on the verge of famine . And this is the result of the farmers-needs research . The nice thing here is that you ca n't see who are the researchers in this picture . That 's when research functions for societies -- you must really live with the people . When you 're in poverty , everything is about survival . It 's about having food . And these two young farmers , they are girls now -- because the parents are dead from HIV and AIDS -- they discuss with a trained agronomist . This is one of the best agronomists in Malawi , Junatambe Kumbira , and he 's discussing what sort of cassava they will plant -- the best converter of sunshine to food that man has found . And they are very , very eagerly interested to get advice , and that 's to survive in poverty . That 's one context . Getting out of poverty . The women told us one thing . " Get us technology . We hate this mortar , to stand hours 
 and hours . Get us a mill so that we can mill our flour , then we will be able to pay for the rest ourselves . " Technology will bring you out of poverty , but there 's a need for a market to get away from poverty . And this woman is very happy now , bringing her products to the market . But she 's very thankful for the public investment in schooling so she can count , and wo n't be cheated when she reaches the market . She wants her kid to be healthy , so she can go to the market and does n't have to stay home . And she wants the infrastructure -- it is nice with a paved road There 's also good with credit . Micro-credits gave her the bicycle , you know . And information will tell her when to go to market with which product . You can do this . I find my experience from 20 years of Africa is that the seemingly impossible is possible . Africa has not done bad . In 50 years they 've gone from a pre-Medieval situation to a very decent 100-year-ago Europe , with a functioning nation and
  state . I would say that sub-Saharan Africa has done best in the world during the last 50 years . Because we do n't consider where they came from . It 's this stupid concept of developing countries which puts us , Argentina and Mozambique together 50 years ago , and says that Mozambique did worse . We have to know a little more about the world . I have a neighbor who knows 200 types of wine . He knows everything . He knows the name of the grape , the temperature and everything . I only know two types of wine -- red and white . ( Laughter ) But my neighbor only knows two types of countries -- industrialized and developing . And I know 200 , I know about the small data . But you can do that . ( Applause ) But I have to get serious . And how do you get serious ? You make a PowerPoint , you know ? ( Laughter ) Homage to the Office package , no ? What is this , what is this , what am I telling ? I 'm telling you that there are many dimensions of development . Everyone wants your pet thi
 ng . If you are in the corporate sector , you love micro-credit . If you are fighting in a non-governmental organization , you love equity between gender . Or if you are a teacher , you 'll love UNESCO , and so on . On the global level , we have to have more than our own thing . We need everything . All these things are important for development , especially when you just get out of poverty and you should go towards welfare . Now , what we need to think about is , what is a goal for development , and what are the means for development ? Let me first grade what are the most important means . Economic growth to me , as a public-health professor , is the most important thing for development , because it explains 80 percent of survival . Governance . To have a government that functions -- that 's what brought California out of the misery of 1850. It was the government which made law function finally . Education , human resources are important . Health is also important , but not that mu
 ch as a mean . Environment is important . Human rights is also important , but it just gets one cross . Now what about goals ? Where are we going toward ? We are not interested in money . Money is not a goal . It 's the best mean , but I give it zero as a goal . Governance , well it 's fun to vote in a little thing , but it 's not a goal . And going to school , that 's not a goal , it 's a mean . Health I give two points . I mean it 's nice to be healthy -- at my age especially -- you can stand here , you 're healthy . And that 's good , it gets two plusses . Environment is very , very crucial . There 's nothing for the grandkid if you do n't save up . But where are the important goals ? Of course , it 's human rights . Human rights is the goal , but it 's not that strong of a mean for achieving development . And culture . Culture is the most important thing , I would say , because that 's what brings joy to life . That 's the value of living . So the seemingly impossible is possibl
 e . Even African countries can achieve this . And I 've shown you the shot where the seemingly impossible is possible . And remember , please remember my main message , which is this : the seemingly impossible is possible . We can have a good world . I showed you the shots , I proved it in the PowerPoint , and I think I will convince you also by culture . ( Laughter ) ( Applause ) Bring me my sword ! Sword swallowing is from ancient India . It 's a cultural expression that for thousands of years has inspired human beings to think beyond the obvious . ( Laughter ) And I will now prove to you that the seemingly impossible is possible by taking this piece of steel -- solid steel -- this is the army bayonet from the Swedish Army , 1850 , in the last year we had war . And it 's all solid steel -- you can hear here . And I 'm going to take this blade of steel , and push it down through my body of blood and flesh , and prove to you that the seemingly impossible is possible . Can I request 
 a moment of absolute silence ? ( Applause ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/102TediSinclairC_OpenArchitech_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/102TediSinclairC_OpenArchitech_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/102TediSinclairC_OpenArchitech_EN.txt.txt
new file mode 100644
index 0000000..143b858
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/102TediSinclairC_OpenArchitech_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff I 'm going to take you on a journey very quickly . To explain the wish , I 'm going to have to take you somewhere which many people have n't been , and that 's around the world . When I was about 24 years old , Kate Store and myself started an organization to get architects and designers involved in humanitarian work . Not only about responding to natural disasters , but involved in systemic issues . We believed that where the resources and expertise are scarce , innovative , sustainable design can really make a difference in people 's lives . So this all began my -- I started my life as an architect , or training as an architect , and I was always interested in socially responsible design , and how you can really make an impact . But when I went to architectural school , it seemed that I was the black sheep in the family . Many architects seemed to think that when you design , you design a jewel , and it 's a jewel that you try and crave for . Whereas I felt that when you desig
 n , you either improve or you create a detriment to the community in which you 're designing in . So you 're not just doing a building for the residents or for the people who are going to use it , but for the community as a whole . And in 1999 , we started by responding to the issue of the housing crisis for returning refugees in Kosovo and I did n't know what I was doing , like I say , mid-20s , and I 'm the , I 'm the Internet generation , so we started a website . We put a call out there , and to my surprise in a couple of months we had hundreds of entries from around the world . That led to a number of prototypes being built and really experimenting with some ideas . Two years later we started doing a project on developing mobile health clinics in sub-Saharan Africa , responding to the HIV/AIDS pandemic . That -- that led to 550 entries from 53 countries . We also have designers from around the world that participate . And we had an exhibit of work that followed that . 2004 was 
 the tipping point for us . We started responding to natural disasters and getting involved in Iran and Bam , also following up on our work in Africa . Working within the United States , most people look at poverty and they see the face of a foreigner , but go live -- I live in Bozeman , Montana -- go up to the north plains on the reservations , or go down to Alabama or Mississippi pre-Katrina , and I could have shown you places that have far worse conditions than many developing countries I 've been to . So we got involved in and worked in inner cities and elsewhere . And then also I will go into some more projects . 2005 Mother Nature kicked our arse . I think we can pretty much assume that 2005 was a horrific year when it comes to natural disasters . And because of the Internet , and because of connections to blogs and so forth , within literally hours of the tsunami , we were already raising funds , getting involved , working with people on the ground . We run from a couple of la
 ptops in the first couple of days , I had 4,000 emails from people needing help . So we began to get involved in projects there , and I 'll talk about some others . And then of course , this year we 've been responding to Katrina , as well as following up on our reconstruction works . This is a brief overview . In 2004 , I really could n't manage the number of people who wanted to help , or the number of requests that I was getting . It was all coming into my laptop and cell phone . So we decided to embrace an open -- basically an open source model of business , that anyone , anywhere in the world , could start a local chapter , and they can get involved in local problems . Because I believe there is no such thing as Utopia . All problems are local . All solutions are local . So , and that means , you know , somebody who is based in , in Mississippi , knows more about Mississippi than I do . So what happened is , we used MeetUp and all these other kind of Internet tools , and we end
 ed up having 40 chapters starting up , thousands of architects in 104 countries . So the , the bullet point -- sorry , I never do a suit , so I knew that I was going to take this off . OK , because I 'm going to do it very quick . So in the past seven years , this is n't just about nonprofit . What it showed me is that there 's a grassroots movement going on of socially responsible designers who really believe that this world has got a lot smaller , and that we have the opportunity -- not the responsibility , but the opportunity -- to really get involved in making change . ( Laughter ) I 'm adding that to my time . So what you do n't know is , we 've got these thousands of designers working around the world , connected basically by a website , and we have a staff of three . By doing something , the fact that nobody told us we could n't do it , we did it . And so there 's something to be said about naivete . So seven years later , we 've developed so that we 've got advocacy , instig
 ation and implementation . We advocate for good design , not only through student workshops and lectures and public forums , op-eds , we have a book on humanitarian work , but also disaster mitigation and dealing with public policy . We can talk about FEMA , but that 's another talk . Instigation , developing ideas with communities and NGOs doing open-source design competitions . Referring , matchmaking with communities and then implementing -- actually going out there and doing the work , because when you invent , it 's never a reality until it 's built . So it 's really important that if we 're designing and trying to create change , we build that change . So here 's a select number of projects . Kosovo . This is Kosovo in '99 . We did an open design competition , like I said . It led to a whole variety of ideas , and this was n't about emergency shelter , but transitional shelter that would last five to 10 years , that would be placed next to the land the resident lived in , and 
 that they would rebuild their own home . This was n't imposing an architecture on a community , this was giving them the tools and , and the space to allow them to rebuild and regrow the way they want to . We have from the sublime to the ridiculous , but they worked . This is an inflatable hemp house . It was built ; it works . This is a shipping container . Built and works . And a whole variety of ideas that not only dealt with architectural building , but also the issues of governance and the idea of creating communities through complex networks . So we 've engaged not just designers , but also , you know , a whole variety of technology-based professionals . Using rubble from destroyed homes to create new homes . Using strawbale construction , creating heat walls . And then something remarkable happened in '99 . We went to Africa , originally to look at the housing issue . Within three days , we realized the problem was not housing ; it was the growing pandemic of HIV/AIDS . And i
 t was n't doctors telling us this ; it was actual villagers that we were staying with . And so we came up with the bright idea that instead of getting people to walk 10 , 15 kilometers to see doctors , you get the doctors to the people . And we started engaging the the medical community . And I thought , you know , we thought we were real bright , you know , sparks -- we 've come up with this great idea , mobile health clinics that can -- widely distributed throughout sub-Saharan Africa . And the community , the medical community there said , " We 've said this for the last decade . We know this . We just do n't know how to show this . " So in a way , we had taken a pre-existing need and shown solutions . And so again , we had a whole variety of ideas that came in . This one I personally love , because the idea that architecture is not just about solutions , but about raising awareness . This is a kenaf clinic . You get seed and you grow it in a plot of land , and then once -- and i
 t grows 14 feet in a month . And on the fourth week , the doctors come and they mow out an area , put a tensile structure on the top and when the doctors have finished treating and seeing patients and villagers , you cut down the clinic and you eat it . It 's an Eat Your Own Clinic . ( Laughter ) So it 's dealing with the fact that if you have AIDS , you also need to have nutrition rates , and the idea that the idea of nutrition is as important as getting anti-retrovirals out there . So you know , this is a serious solution . This one I love . The idea is it 's not just a clinic -- it 's a community center . This looked at setting up trade routes and economic engines within the community , so it can be a self-sustaining project . Every one of these projects is sustainable . That 's not because I 'm a tree-hugging green person . It 's because when you live on four dollars a day , you 're living on survival and you have to be sustainable . You have to know where your energy is coming 
 from . You have to know where your resource is coming from . And you have to keep the maintenance down . So this is about getting an economic engine , and then at night it turns into a movie theater . So it 's not an AIDS clinic . It 's a community center . So you can see ideas . And these ideas developed into prototypes , and they were eventually built . And currently as of this year , there are clinics rolling out in Nigeria and Kenya . From that we also developed Siyathemba , which was a project -- the community came to us and said , the problem is that the girls do n't have education . And we 're working in an area where young women between the ages of 16 and 24 have a 50 percent HIV/AIDS rate . And that 's not because they 're promiscuous , it 's because there 's no knowledge . And so we decided to look at the idea of sports and create a youth sports center that doubled as an HIV/AIDS outreach center , and the coaches of the girls ' team were also trained doctors . So that ther
 e would be a very slow way of developing kind of confidence in health care . And we picked nine finalists , and then those nine finalists were distributed throughout the entire region , and then the community picked their design . They said , this is our design , because it 's not only about engaging a community , it 's about empowering a community and about getting them to be a part of the rebuilding process . So the winning design is here , and then of course , we actually go and work with the community and the clients . So this is the designer . He 's out there working with the first ever women 's soccer team in Kwa-Zulu Natal , Siyathemba , and they can tell it better . Video : Well , my name is Sisi , because I work at the African center . I 'm a consultant and I 'm also the national football player for South Africa , Bafana Bafana , and I also play in the Vodacom League for the team called Tembisa , which has now changed to Siyathemba . This is our home ground . Cameron Sincla
 ir : So I 'm going to show that later because I 'm running out of time . I can see Chris looking at me slyly . This was a connection , just a meeting with somebody who wanted to develop Africa 's first telemedicine center , in Tanzania . And we met , literally , a couple of months ago . We 've already developed a design , and the team is over there , working in partnership . This was a matchmaking , thanks to a couple of TEDsters : [ unclear ] Cheryl Heller and Andrew Zolli , who connected me with this amazing African woman . And we start construction in June , and it will be opened by TEDGlobal . So when you come to TEDGlobal , you can check it out . But what we 're known probably most for is dealing with disasters and development , and we 've been involved in a lot of issues , such as the tsunami and also things like Hurricane Katrina . This is a 370 dollar shelter that can be easily assembled . This is a community design . A community-designed community center . And what that mea
 ns is we actually live and work with the community , and they 're part of the design process . The kids actually get involved in mapping out where the the community center should be , and then eventually , the community is actually , through skills training , end up building the building with us . Here is another school . This is what the U. N. gave these guys for six months -- 12 plastic tarps . This was in August . This was the replacement , and it 's supposed to last for two years . When the rain comes down , you ca n't hear a thing , and in the summer it 's about 140 degrees inside . So we said , if the rain 's coming down , let 's get fresh water . So every one of our schools have rain water collection systems , very low cost . A class , three classrooms and rainwater collection is five thousand dollars . This was raised by hot chocolate sales in Atlanta . It 's built by the parents of the kids . The kids are out there on-site , building the buildings . And it opened a couple o
 f weeks ago , and there 's 600 kids that are now using the schools . ( Applause ) So , disaster hits home . We 've see the bad stories on CNN and Fox and all that , but we do n't see the good stories . Here is a community that got together and they said no to wait , to waiting . They formed a partnership , a diverse partnership of players to actually map out East Biloxi , to figure out who is getting involved . We 've had 1,500 volunteers rebuilding , rehabbing homes . Figuring out what FEMA regulations are , not waiting for them to dictate to us how you should rebuild . Working with residents , getting out -- them out of their homes , so they do n't get ill . This is what they 're cleaning up on their own . Designing housing . This house is going to go in , in a couple of weeks . This is a rehabbed home , done in four days. This is a utility room for a woman who is on a walker . She 's 70 years old . This is what FEMA gave her . 600 bucks , happened two days ago . We put together v
 ery quickly a washroom . It 's built , it 's running and she just started a business today , where she 's washing other peoples clothes . This is Shandra and the Calhouns . They 're photographers who have documented the Lower Ninth for the last 40 years . That was their home , and these are the photographs they took . And we 're helping , working with them to create a new building . Projects we 've done . Projects we 've been a part of , support . Why do n't aid agencies do this ? This is the U. N. tent . This is the new U. N. tent , just introduced this year . Quick to assemble . It 's got a flap , that 's the invention . It took 20 years to design this and get it implemented in the field . I was 12 years old . There 's a problem here . Luckily , we 're not alone . There are hundreds and hundreds and hundreds and hundreds and hundreds of architects and designers and inventors around the world that are getting involved in humanitarian work . More hemp houses -- it 's a theme in Japa
 n apparently . I 'm not sure what they 're smoking . This is a grip clip designed by somebody who said , all you need is some way to attach membrane structures to physical support beams . This guy , designed for NASA -- is now doing housing . I 'm going to whip through this quickly , because I know I 've got only a couple of minutes . So this is all done in the last two years . I showed you something that took 20 years to do . And this is just a selection of things that got happened in -- that were built in the last couple of years . From Brazil to India , Mexico , Alabama , China , Israel , Palestine , Vietnam . The average age of a designer who gets involved in this project is 32 -- that 's how old I am . So it 's a young -- I just have to stop here , because Arup is in the room and this is the best-designed toilet in the world . If you 're ever , ever in India , go use this toilet . ( Laughter ) Chris Luebkeman will tell you why . I 'm sure that 's how he wanted to spend the part
 y , but -- but the future is not going to be the sky-scraping cities of New York , but this . And when you look at this , you see crisis . What I see is many , many inventors . One billion people live in abject poverty . We hear about them all the time . Four billion live in growing but fragile economies . One in seven live in unplanned settlements . If we do nothing about the housing crisis that 's about to happen , in 20 years , one in three people will live in an unplanned settlement or a refugee camp . Look left , look right : one of you will be there . How do we improve the living standards of five billion people ? With 10 million solutions . So I wish to develop a community that actively embraces innovative and sustainable design to improve the living conditions for everyone . Chris Anderson : Wait a second . Wait a second . That 's your wish ? CS : That 's my wish . CA : That 's his wish ! ( Applause ) We started Architecture for Humanity with 700 dollars and a website . So C
 hris somehow decided to give me 100,000 dollars . So why not this many people ? Open-source architecture is the way to go . You have a diverse community of participants -- and we 're not just talking about inventors and designers , but we 're talking about the funding model . My role is not as a designer ; it 's a conduit between the design world and the humanitarian world . And what we need is something that replicates me globally , because I have n't slept in seven years . ( Laughter ) Secondly , what will this thing be ? Designers want to respond to issues of humanitarian crisis , but they do n't want some company in the West taking their idea and basically profiting from it . So Creative Commons has developed the developing nations license . And what that means that a designer can -- the Siyathemba project I showed was the first ever building to have a Creative Commons license on it . As soon as that is built , anyone in Africa or any developing nation can take the construction 
 documents and replicate it for free . ( Applause ) So why not allow designers the opportunity to do this , but still protecting their rights , here ? We want to have a community where you can upload ideas , and those ideas can be tested in earthquake , in flood , in all sorts of austere environments . The reason that 's important is I do n't want to wait for the next Katrina to find out if my house works . That 's too late . We need to do it now . So doing that globally . And I want this whole thing to work multi-lingually . When you look at the face of an architect , most people think a gray-haired white guy . I do n't see that . I see the face of the world . So I want everyone from all over the planet , to be able to be a part of this design and development . The idea of needs-based competitions -- X-Prize for the other 98 percent , if you want to call it that . We also want to look at ways of matchmaking and putting funding partners together . And the idea of integrating manufact
 urers -- fab labs in every country . When I hear about the 100 dollar laptop and it 's going to educate every child , educate every designer in the world . Put one in every favela , every slum settlement , because you know what , innovation will happen . And I need to know that . It 's called the leap-back . We talk about leapfrog technologies . I write with Worldchanging , and the one thing we 've been talking about is , I learn more on the ground than I 've ever learned here . So let 's take those ideas , adapt them and we can use them . These ideas are supposed to have adaptable , they 're allowed to be -- they should have the potential for evolution , they should be developed by every nation on the world and useful for every nation on the world . What will it take ? There should be a sheet . I do n't have time to read this , because I 'm going to be yanked off . CA : Just leave it up there for a sec . CS : Well , what will it take ? You guys are smart . So it 's going to take a 
 lot of computing power , because I want this to -- I want the idea that any laptop anywhere in the world can plug into the system and be able to not only participate in developing these designs , but utilize the designs . Also , a process of reviewing the designs . I want every Arup engineer in the world to check and make sure that we 're doing stuff that 's standing , because those guys are the best in the world . Plug . And so you know , I want these -- and I just should note , I have two laptops and one of them there , is there and that has 3000 designs on it . If I drop that laptop , what happens ? So it 's important to have these proven ideas put up there , easy to use , easy to get ahold of . My mom once said , there 's nothing worse than being all mouth and no trousers . ( Laughing ) I 'm fed up of talking about making change . You only make it by doing it . We 've changed FEMA guidelines . We 've changed public policy . We 've changed international response -- based on build
 ing things . So for me , it 's important that we create a real conduit for innovation , and that it 's free innovation . Think of free culture -- this is free innovation . Somebody said this a couple of years back . I will give points for those who know it , I think the man was maybe 25 years too early , so let 's do it . Thank you . ( Applause ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/92TediAndersonEt_NuclearEnergy_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/92TediAndersonEt_NuclearEnergy_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/92TediAndersonEt_NuclearEnergy_EN.txt.txt
new file mode 100644
index 0000000..2f9e523
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/92TediAndersonEt_NuclearEnergy_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Chris Anderson : We 're having a debate . The debate is over the proposition " What the world needs now is nuclear energy " -- true or false ? And before we have the debate , I 'd like to actually take a show of hands -- on balance , right now , are you for or against this ? So those who are " yes , " raise your hand . " For . " Okay , hands down . Those who are " against , " raise your hands . Okay , I 'm reading that at about 75-25 in favor at the start . Which means we 're going to take a vote at the end and see how that shifts , if at all . So here 's the format : They 're going to have six minutes each , and then after one little , quick exchange between them , I want two people on each side of this debate in the audience to have 30 seconds to make one short , crisp , pungent , powerful point . So , in favor of the proposition , possibly shockingly , is one of , truly , the founders of the environmental movement , a long-standing TEDster , the founder of the Whole Earth Cat
 alog , someone we all know and love , Stewart Brand . Stewart Brand : Whoa . ( Applause ) The saying is that with climate , those who know the most are the most worried . With nuclear , those who know the most are the least worried . A classic example is James Hansen , a NASA climatologist pushing for 350 parts per million carbon dioxide in the atmosphere . He came out with a wonderful book recently called " Storms of My Grandchildren . " And Hansen is hard over for nuclear power , as are most climatologists who are engaging this issue seriously . This is the design situation : a planet that is facing climate change and is now half urban . Look at the client base for this . Five out of six of us live in the developing world . We are moving to cities . We are moving up in the world . And we are educating our kids , having fewer kids , basically good news all around . But we move to cities , toward the bright lights , and one of the things that is there that we want , besides jobs , i
 s electricity . And if it is n't easily gotten , we 'll go ahead and steal it . This is one of the most desired things by poor people all over the world , in the cities and in the countryside . Electricity for cities , at its best , is what 's called baseload electricity . That 's where it is on all the time . And so far there are only three major sources of that -- coal and gas , hydro-electric , which in most places is maxed-out -- and nuclear . I would love to have something in the fourth place here , but in terms of constant , clean , scalable energy , [ solar ] and wind and the other renewables are n't there yet because they 're inconstant . Nuclear is and has been for 40 years . Now , from an environmental standpoint , the main thing you want to look at is what happens to the waste from nuclear and from coal , the two major sources of electricity . If all of your electricity in your lifetime came from nuclear , the waste from that lifetime of electricity would go in a Coke can
  -- a pretty heavy Coke can , about two pounds . But one day of coal adds up to one hell of a lot of carbon dioxide in a normal one-gigawatt coal-fired plant . Then what happens to the waste ? The nuclear waste typically goes into a dry cask storage out back of the parking lot at the reactor site because most places do n't have underground storage yet . It 's just as well , because it can stay where it is . While the carbon dioxide , vast quantities of it , gigatons , goes into the atmosphere where we ca n't get it back , yet , and where it is causing the problems that we 're most concerned about . So when you add up the greenhouse gases in the lifetime of these various energy sources , nuclear is down there with wind and hydro , below solar and way below , obviously , all the fossil fuels . Wind is wonderful ; I love wind . I love being around these big wind generators . But one of the things we 're discovering is that wind , like solar , is an actually relatively dilute source of 
 energy . And so it takes a very large footprint on the land , a very large footprint in terms of materials , five to 10 times what you 'd use for nuclear , and typically to get one gigawatt of electricity is on the order of 250 sq . mi . of wind farm . In places like Denmark and Germany , they 've maxed out on wind already . They 've run out of good sites . The power lines are getting overloaded . And you peak out . Likewise , with solar , especially here in California , we 're discovering that the 80 solar farm schemes that are going forward want to basically bulldoze 1,000 sq . mi . of southern California desert . Well , as an environmentalist , we would rather that did n't happen . It 's okay on frapped-out agricultural land . Solar 's wonderful on rooftops . But out in the landscape , one gigawatt is on the order of 50 sq . mi . of bulldozed desert . When you add all these things up -- Saul Griffith did the numbers and figured out what it would take to get 13 clean terawatts of 
 energy from wind , solar and biofuels , and that area would be roughly the size the United States , an area he refers to as " Renewistan . " A guy who 's added all this up very well is David Mackay , a physicist in England , and in his wonderful book , " Sustainable Energy , " among other things , he says , " I 'm not trying to be pro-nuclear . I 'm just pro-arithmetic . " ( Laughter ) In terms of weapons , the best disarmament tool so far is nuclear energy . We have been taking down the Russian warheads , turning it into electricity . 10 percent of American electricity comes from decommissioned warheads . We have n't even started the American stockpile . I think of most interest to a TED audience would be the new generation of reactors that are very small , down around 10 to 125 megawatts . This is one from Toshiba . Here 's one that the Russians are already building that floats on a barge . And that would be very interesting in the developing world . Typically , these things are p
 ut in the ground . They 're referred to as nuclear batteries . They 're incredibly safe , weapons proliferation-proof and all the rest of it . Here is a commercial version from New Mexico called the Hyperion , and another one from Oregon called NuScale . Babcock & Wilcox that make nuclear reactors ... here 's an integral fast reactor . Thorium reactor that Nathan Myhrvold 's involved in . The governments of the world are going to have to decide that coal needs to be made expensive , and these will go ahead . And here 's the future . ( Applause ) CA : Okay . Okay . ( Applause ) So arguing against , a man who 's been at the nitty , gritty heart of the energy debate and the climate change debate for years . In 2000 , he discovered that soot was probably the second leading cause of global warming , after CO2 . His team have been making detailed calculations of the relative impacts of different energy sources . His first time at TED , possibly a disadvantage -- we shall see -- from Stanf
 ord , Professor Mark Jacobson . Good luck . Mark Jacobson : Thank you . ( Applause ) So my premise here is that nuclear energy puts out more carbon dioxide , puts out more air pollutants , enhances mortality more and takes longer to put up than real renewable energy systems , namely wind , solar , geothermal power , hydro-tidal wave power . And it also enhances nuclear weapons proliferation . So let 's just start by looking at the CO2 emissions from the life cycle . CO2e emissions are equivalent emissions of all the greenhouse gases and particles that cause warming , and converted to CO2 . And if you look , wind and concentrated solar have the lowest CO2 emissions , if you look at the graph . Nuclear -- there are two bars here . One is a low estimate , and one is a high estimate . The low estimate is the nuclear energy industry estimate of nuclear . The high is the average of 103 scientific , peer-reviewed studies . And this is just the CO2 from the life cycle . If we look at the de
 lays , it takes between 10 and 19 years to put up a nuclear power plant from planning to operation . This includes about three and a half to six years for a site permit . and another two and a half to four years for a construction permit and issue , and then four to nine years for actual construction . And in China , right now , they 're putting up five gigawatts of nuclear . And the average , just for the construction time of these , is 7.1 years on top of any planning times . While you 're waiting around for your nuclear , you have to run the regular electric power grid , which is mostly coal in the United States and around the world . And the chart here shows the difference between the emissions from the regular grid , resulting if you use nuclear , or anything else , versus wind , CSP or photovoltaics . Wind takes about two to five years on average , same as concentrated solar and photovoltaics . So the difference is the opportunity cost of using nuclear versus wind , or somethi
 ng else . So if you add these two together , alone , you can see a separation that nuclear puts out at least nine to 17 times more CO2 equivalent emissions than wind energy . And this does n't even account for the footprint on the ground . If you look at the air pollution health effects , this is the number of deaths per year in 2020 just from vehicle exhaust . Let 's say we converted all the vehicles in the United States to battery electric vehicles , hydrogen fuel cell vehicles or flex fuel vehicles run on E85 . Well , right now in the United States , 50 to 100,000 people die per year from air pollution , and vehicles are about 25,000 of those . In 2020 , the number will go down to 15,000 due to improvements . And so , on the right , you see gasoline emissions , the death rates of 2020. If you go to corn or cellulosic ethanol , you 'd actually increase the death rate slightly . If you go to nuclear , you do get a big reduction , but it 's not as much as with wind and/or concentrat
 ed solar . Now if you consider the fact that nuclear weapons proliferation is associated with nuclear energy proliferation , because we know for example , India and Pakistan developed nuclear weapons secretly by enriching uranium in nuclear energy facilities . North Korea did that to some extent . Iran is doing that right now . And Venezuela would be doing it if they started with their nuclear energy facilities . If you do a large scale expansion of nuclear energy across the world , and as a result there was just one nuclear bomb created that was used to destroy a city such as Mumbai or some other big city , megacity , the additional death rates due to this averaged over 30 years and scaled to the population of the U. S. would be this . So , do we need this ? The next thing is : What about the footprint ? Stewart mentioned the footprint . Actually , the footprint on the ground for wind is by far the smallest of any energy source in the world . That , because the footprint , as you c
 an see , is just the pole touching the ground . And you can power the entire U. S. vehicle fleet with 73,000 to 145,000 five-megawatt wind turbines . That would take between one and three sq . km . of footprint on the ground , entirely . The spacing is something else . That 's the footprint that 's always being confused . People confuse footprint with spacing . As you can see from these pictures , the spacing between can be used for multiple purposes including agricultural land , range land or open space . Over the ocean , it 's not even land . Now if we look at nuclear -- ( Laughter ) With nuclear , what do we have ? We have facilities around there . You also have a buffer zone that 's 17 sq . km . And you have the uranium mining that you have to deal with . Now if we go to the area , lots is worse than nuclear or wind . For example , cellulosic ethanol , to power the entire U. S. vehicle fleet , this is how much land you would need . That 's cellulosic , second generation biofuels
  from prairie grass . Here 's corn ethanol . It 's smaller . This is based on ranges from data , but if you look at nuclear , it would be the size of Rhode Island to power the U. S. vehicle fleet . For wind , there 's a larger area , but much smaller footprint . And of course , with wind , you could put it all over the east coast , offshore theoretically , or you can split it up . And now , if you go back to looking at geothermal , it 's even smaller than both , and solar is slightly larger than the nuclear spacing , but it 's still pretty small . And this is to power the entire U. S. vehicle fleet . To power the entire world with 50 percent wind , you would need about one percent of world land . Matching the reliability , base load is actually irrelevant . We want to match the hour-by-hour power supply . You can do that by combining renewables . This is from real data in California , looking at wind data and solar data . And it considers just using existing hydro to match the hour-
 by-hour power demand . Here are the world wind resources . There 's 5 to 10 times more wind available worldwide than we need for all the world . So then the finally ranking . And one last slide I just want to show : this is the choice . You can either have wind or nuclear . If you use wind , you guarantee ice will last . Nuclear , the time lag alone will allow the Arctic to melt and other places to melt more . And we can guarantee a clean , blue sky or an uncertain future with nuclear power . ( Applause ) CA : All right . So while they 're having their comebacks on each other -- and yours is slightly short because you slightly overran -- I need two people from either side . So if you 're for this , if you 're for nuclear power , put up two hands . If you 're against , put up one . And I want two of each for the mics . Now then , you guys have -- you have a minute comeback on him to pick up a point he said , challenge it , whatever . SB : I think a point of difference we 're having ,
  Mark , has to do with weapons and energy . These diagrams that show that nuclear is somehow putting out a lot of greenhouse gases -- a lot of those studies will include , " Well of course war will be inevitable and therefore we 'll have cities burning and stuff like that , " which is kind of finessing it a little bit , I think . The reality is that there 's , what , 21 nations that have nuclear power ? Of those , seven have nuclear weapons . In every case , they got the weapons before they got the nuclear power . There are two nations , North Korea and Israel , that have nuclear weapons and do n't have nuclear power at all . The places that we would most like to have really clean energy occur are China , India , Europe , North America , all of which have sorted out their situation in relation to nuclear weapons . So that leaves a couple of places like Iran , maybe Venezuela , that you would like to have very close surveillance of anything that goes on with fissile stuff . Pushing a
 head with nuclear power will mean we really know where all of the fissile material is , and we can move toward zero weapons left , once we know all that . CA : Mark , 30 seconds , either on that or on anything Stewart said . MJ : Well we know India and Pakistan had nuclear energy first , and then they developed nuclear weapons secretly in the factories . So the other thing is , we do n't need nuclear energy . There 's plenty of solar and wind . You can make it reliable , as I showed with that diagram . That 's from real data . And this is an ongoing research . This is not rocket science . Solving the world 's problems can be done , if you 're really put your mind to it and use clean , renewable energy . There 's absolutely no need for nuclear power . ( Applause ) CA : We need someone for . Rod Beckstrom : Thank you Chris . I 'm Rod Beckstrom , CEO of ICANN . I 've been involved in global warming policy since 1994 , when I joined the board of Environmental Defense Fund that was one o
 f the crafters of the Kyoto Protocol . And I want to support Stewart Brand 's position . I 've come around in the last 10 years . I used to be against nuclear power . I 'm now supporting Stewart 's position , softly , from a risk-management standpoint , agreeing that the risks of overheating the planet outweigh the risk of nuclear incident , which certainly is possible and is a very real problem . However , I think there may be a win-win solution here where both parties can win this debate , and that is , we face a situation where it 's carbon caps on this planet or die . And in the United States Senate , we need bipartisan support -- only one or two votes are needed -- to move global warming through the Senate , and this room can help . So if we get that through , then Mark will solve these problems . Thanks Chris . CA : Thank you Rod Beckstrom . Against . David Fanton : Hi , I 'm David Fanton . I just want to say a couple quick things . The first is : be aware of the propaganda . 
 The propaganda from the industry has been very , very strong . And we have not had the other side of the argument fully aired so that people can draw their own conclusions . Be very aware of the propaganda . Secondly , think about this . If we build all these nuclear power plants , all that waste is going to be on hundreds , if not thousands , of trucks and trains , moving through this country every day . Tell me they 're not going to have accidents . Tell me that those accidents are n't going to put material into the environment that is poisonous for hundreds of thousands of years And then tell me that each and every one of those trucks and trains is n't a potential terrorist target . CA : Thank you . For . Anyone else for ? Go . Alex : Hi , I 'm Alex . I just wanted to say , I 'm , first of all , renewable energy 's biggest fan . I 've got solar PV on my roof . I 've got a hydro conversion at a watermill that I own . And I 'm , you know , very much " pro " that kind of stuff . How
 ever , there 's a basic arithmetic problem here . The capability of the sun shining , the wind blowing and the rain falling , simply is n't enough to add up . So if we want to keep the lights on , we actually need a solution which is going to keep generating all of the time . I campaigned against nuclear weapons in the 80s , and I continue to do so now . But we 've got an opportunity to recycle them into something more useful that enables us to get energy all of the time . And , ultimately , the arithmetic problem is n't going to go away . We 're not going to get enough energy from renewables alone . We need a solution that generates all of the time . If we 're going to keep the lights on , nuclear is that solution . CA : Thank you . Anyone else against ? Man : The last person who was in favor made the premise that we do n't have enough alternative renewable resources . And our " against " proponent up here made it clear that we actually do . And so the fallacy that we need this res
 ource and we can actually make it in a time frame that is meaningful is not possible . I will also add one other thing . Ray Kurzweil and all the other talks -- we know that the stick is going up exponentially . So you ca n't look at state-of-the-art technologies in renewables and say , " That 's all we have . " Because five years from now , it will blow you away what we 'll actually have as alternatives to this horrible , disastrous nuclear power . CA : Point well made . Thank you . ( Applause ) So each of you has really just a couple sentences -- 30 seconds each to sum up . Your final pitch , Stewart . SB : I loved your " It all balances out " chart that you had there . It was a sunny day and a windy night . And just now in England they had a cold spell . All of the wind in the entire country shut down for a week . None of those things were stirring . And as usual , they had to buy nuclear power from France . Two gigawatts comes through the Chunnel . This keeps happening . I used 
 to worry about the 10,000 year factor . And the fact is , we 're going to use the nuclear waste we have for fuel in the fourth generation of reactors that are coming along . And especially the small reactors need to go forward . I heard from Nathan Myhrvold -- and I think here 's the action point -- it 'll take an act of Congress to make the Nuclear Regulatory Commission start moving quickly on these small reactors , which we need very much , here and in the world . ( Applause ) MJ : So we 've analyzed the hour-by-hour power demand and supply , looking at solar , wind , using data for California . And you can match that demand , hour-by-hour , for the whole year almost . Now , with regard to the resources , we 've developed the first wind map of the world , from data alone , at 80 meters . We know what the resources are . You can cover 15 percent . 15 percent of the entire U. S. has wind at fast-enough speeds to be cost-competitive . And there 's much more solar than there is wind .
  There 's plenty of resource . You can make it reliable . CA : Okay . So , thank you , Mark . ( Applause ) So if you were in Palm Springs ... ( Laughter ) ( Applause ) Shameless . Shameless . Shameless . ( Applause ) So , people of the TED community , I put it to you that what the world needs now is nuclear energy . All those in favor , raise your hands . ( Shouts ) And all those against . Ooooh . Now that is -- my take on that ... Just put up ... Hands up , people who changed their minds during the debate , who voted differently . Those of you who changed your mind in favor of " for " put your hands up . Okay . So here 's the read on it . Both people won supporters , but on my count , the mood of the TED community shifted from about 75-25 to about 65-35 in favor , in favor . You both won . I congratulate both of you . Thank you for that . ( Applause ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/93TediBelcherA_Batteries_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/93TediBelcherA_Batteries_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/93TediBelcherA_Batteries_EN.txt.txt
new file mode 100644
index 0000000..acc3dbe
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/93TediBelcherA_Batteries_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff I thought I would talk a little bit about how nature makes materials . I brought along with me an abalone shell . This abalone shell is a biocomposite material that 's 98 percent by mass calcium carbonate and two percent by mass protein . Yet , it 's 3,000 times tougher than its geological counterpart . And a lot of people might use structures like abalone shells , like chalk . I 've been fascinated by how nature makes materials , and there 's a lot of sequence to how they do such an exquisite job . Part of it is that these materials are macroscopic in structure , but they 're formed at the nanoscale . They 're formed at the nanoscale , and they use proteins that are coded by the genetic level that allow them to build these really exquisite structures . So something I think is very fascinating is what if you could give life to non-living structures , like batteries and like solar cells ? What if they had some of the same capabilities that an abalone shell did , in terms of being
  able to build really exquisite structures at room temperature and room pressure , using non-toxic chemicals and adding no toxic materials back into the environment ? So that 's the vision that I 've been thinking about . And so what if you could grow a battery in a petri dish ? Or , what if you could give genetic information to a battery so that it could actually become better as a function of time , and do so in an environmentally friendly way ? And so , going back to this abalone shell , besides being nano-structured , one thing that 's fascinating , is when a male and a female abalone get together , they pass on the genetic information that says , " This is how to build an exquisite material . Here 's how to do it at room temperature and pressure , using non-toxic materials . " Same with diatoms , which are shone right here , which are glasseous structures . Every time the diatoms replicate , they give the genetic information that says , " Here 's how to build glass in the ocean
  that 's perfectly nano-structured . And you can do it the same , over and over again . " So what if you could do the same thing with a solar cell or a battery ? I like to say my favorite biomaterial is my four year-old . But anyone who 's ever had , or knows , small children knows they 're incredibly complex organisms . And so if you wanted to convince them to do something they do n't want to do , it 's very difficult . So when we think about future technologies , we actually think of using bacteria and virus , simple organisms . Can you convince them to work with a new tool box , so that they can build a structure that will be important to me ? Also , we think about future technologies . We start with the beginning of Earth . Basically , it took a billion years to have life on Earth . And very rapidly , they became multi-cellular , they could replicate , they could use photosynthesis as a way of getting their energy source . But it was n't until about 500 million years ago -- duri
 ng the Cambrian geologic time period -- that organisms in the ocean started making hard materials . Before that they were all soft , fluffy structures . And it was during this time that there was increased calcium and iron and silicon in the environment . And organisms learned how to make hard materials . And so that 's what I would like be able to do -- convince biology to work with the rest of the periodic table . Now if you look at biology , there 's many structures like DNA and antibodies and proteins and ribosomes that you 've heard about that are already nano-structured . So nature already gives us really exquisite structures on the nanoscale . What if we could harness them and convince them to not be an antibody that does something like HIV ? But what if we could convince them to build a solar cell for us ? So here are some examples : these are some natural shells . There are natural biological materials . The abalone shell here -- and if you fracture it , you can look at the
  fact that it 's nano-structured . There 's diatoms made out of SIO2 , and they 're magnetotactic bacteria that make small , single-domain magnets used for navigation . What all these have in common is these materials are structured at the nanoscale , and they have a DNA sequence that codes for a protein sequence , that gives them the blueprint to be able to build these really wonderful structures . Now , going back to the abalone shell , the abalone makes this shell by having these proteins . These proteins are very negatively charged . And they can pull calcium out of the environment , put down a layer of calcium and then carbonate , calcium and carbonate . It has the chemical sequences of amino acids which says , " This is how to build the structure . Here 's the DNA sequence , here 's the protein sequence in order to do it . " And so an interesting idea is , what if you could take any material that you wanted , or any element on the periodic table , and find its corresponding DN
 A sequence , then code it for a corresponding protein sequence to build a structure , but not build an abalone shell -- build something that , through nature , it has never had the opportunity to work with yet . And so here 's the periodic table . And I absolutely love the periodic table . Every year for the incoming freshman class at MIT , I have a periodic table made that says , " Welcome to MIT . Now you 're in your element . " And you flip it over , and it 's the amino acids with the PH at which they have different charges . And so I give this out to thousands of people . And I know it says MIT , and this is Caltech , but I have a couple extra if people want it . And I was really fortunate to have President Obama visit my lab this year on his visit to MIT , and I really wanted to give him a periodic table . So I stayed up at night , and I talked to my husband , " How do I give President Obama a periodic table ? What if he says , 'Oh , I already have one , ' or , 'I 've already m
 emorized it ' ? " And so he came to visit my lab and looked around -- it was a great visit . And then afterward , I said , " Sir , I want to give you the periodic table in case you 're ever in a bind and need to calculate molecular weight . " And I thought molecular weight sounded much less nerdy than molar mass . And so he looked at it , and he said , " Thank you . I 'll look at it periodically . " ( Laughter ) ( Applause ) And later in a lecture that he gave on clean energy , he pulled it out and said , " And people at MIT , they give out periodic tables . " So basically what I did n't tell you is that about 500 million years ago , organisms starter making materials , but it took them about 50 million years to get good at it . It took them about 50 million years to learn how to perfect how to make that abalone shell . And that 's a hard sell to a graduate student . " I have this great project -- 50 million years . " And so we had to develop a way of trying to do this more rapidly 
 . And so we use a virus that 's a non-toxic virus called M13 bacteriophage that 's job is to infect bacteria . Well it has a simple DNA structure that you can go in and cut and paste additional DNA sequences into it . And by doing that , it allows the virus to express random protein sequences . And this is pretty easy biotechnology . And you could basically do this a billion times . And so you can go in and have a billion different viruses that are all genetically identical , but they differ from each other based on their tips , on one sequence that codes for one protein . Now if you take all billion viruses , and you can put them in one drop of liquid , you can force them to interact with anything you want on the periodic table . And through a process of selection evolution , you can pull one of a billion that does something that you 'd like it to do , like grow a battery or grow a solar cell . So basically , viruses ca n't replicate themselves , they need a host . Once you find th
 at one out of a billion , you infect it into a bacteria , and you make millions and billions of copies of that particular sequence . And so the other thing that 's beautiful about biology is that biology gives you really exquisite structures with nice link scales . And these viruses are long and skinny , and we can get them to express the ability to grow something like semiconductors or materials for batteries . Now this is a high-powered battery that we grew in my lab . We engineered a virus to pick up carbon nanotubes . So one part of the virus grabs a carbon nanotube . The other part of the virus has a sequence that can grow an electrode material for a battery . And then it wires itself to the current collector . And so through a process of selection evolution , we went from having a virus that made a crummy battery to a virus that made a good battery to a virus that made a record-breaking , high-powered battery that 's all made at room temperature , basically at the bench top . 
 And that battery went to the White House for a press conference . I brought it here . You can see it in this case -- that 's lighting this LED . Now if we could scale this , you could actually use it to run your Prius , which is my dream -- to be able to drive a virus-powered car . But it 's basically -- you can pull one out of a billion . You can make lots of amplifications to it . Basically , you make an amplification in the lab . And then you get it to self-assemble into a structure like a battery . We 're able to do this also with catalysis . This is the example of photocatalytic splitting of water . And what we 've been able to do is engineer a virus to basically take dye absorbing molecules and line them up on the surface of the virus so it acts as an antenna , and you get an energy transfer across the virus . And then we give it a second gene to grow an inorganic material that can be used to split water into oxygen and hydrogen , that can be used for clean fuels . And I broug
 ht an example with me of that today . My students promised me it would work . These are virus-assembled nanowires . When you shine light on them , you can see them bubbling . In this case , you 're seeing oxygen bubbles come out . And basically by controlling the genes , you can control multiple materials to improve your device performance . The last example are solar cells . You can also do this with solar cells . We 've been able to engineer viruses to pick up carbon nanotubes and then grow titanium dioxide around them -- and use as a way of getting electrons through the device . And what we 've found is that , through genetic engineering , we can actually increase the efficiencies of these solar cells to record numbers for these types of dye-sensitized systems . And I brought one of those as well that you can play around with outside afterward . So this is a virus-based solar cell . Through evolution and selection , we took it from an eight percent efficiency solar cell to an 11 
 percent efficiency solar cell . So I hope that I 've convinced you that there 's a lot of great , interesting things to be learned about how nature makes materials -- and taking it to the next step to see if you can force , or whether you can take advantage of how nature makes materials , to make things that nature has n't yet dreamed of making . Thank you . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/94TediGellMannM_Lang_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/94TediGellMannM_Lang_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/94TediGellMannM_Lang_EN.txt.txt
new file mode 100644
index 0000000..cc7b196
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/94TediGellMannM_Lang_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Well , I 'm involved in other things besides physics . In fact , mostly now in other things . One thing is distant relationships among human languages . And the professional , historical linguists in the US and in Western Europe mostly try to stay away from any long-distance relationships ; big groupings , groupings that go back a long time , longer than the familiar families . They do n't like that ; they think it 's crank . I do n't think it 's crank . And there are some brilliant linguists , mostly Russians , who are working on that at Santa Fe Institute and in Moscow , and I would love to see where that leads . Does it really lead to a single ancestor some 20 , 25,000 years ago ? And what if we go back beyond that single ancestor , when there was presumably a competition among many languages ? How far back does that go ? How far back does modern language go ? How many tens of thousands of years does it go back ? Chris Anderson : Do you have a hunch or a hope for what the ans
 wer to that is ? Murray Gell-Mann : Well , I would guess that modern language must be older than the cave paintings and cave engravings and cave sculptures and dance steps in the soft clay in the caves in Western Europe in the Aurignacian Period some 35,000 years ago , or earlier . I ca n't believe they did all those things and did n't also have a modern language . So I would guess that the actual origin goes back at least that far and maybe further . But that does n't mean that all , or many , or most of today 's attested languages could n't descend perhaps from one that 's much younger than that , like say 20,000 years , or something of that kind . It 's what we call a bottleneck . CA : Well , Philip Anderson may have been right . You may just know more about everything than anyone . So it 's been an honor . Thank you Murray Gell-Mann . ( Applause ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/95TediJakubowskiM_OpenTech_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/95TediJakubowskiM_OpenTech_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/95TediJakubowskiM_OpenTech_EN.txt.txt
new file mode 100644
index 0000000..a776cfa
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/95TediJakubowskiM_OpenTech_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Hi , my name is Marcin -- farmer , technologist . I was born in Poland , now in the U. S. I started a group called Open Source Ecology . We 've identified the 50 most important machines that we think it takes for modern life to exist -- things from tractors , bread ovens , circuit makers . Then we set out to create an open source , DIY , do it yourself version that anyone can build and maintain at a fraction of the cost . We call this the Global Village Construction Set . So let me tell you a story . So I finished my 20s with a Ph. D. in fusion energy , and I discovered I was useless . I had no practical skills . The world presented me with options , and I took them . I guess you can call it the consumer lifestyle . So I started a farm in Missouri and learned about the economics of farming . I bought a tractor -- then it broke . I paid to get it repaired -- then it broke again . Then pretty soon I was broke too . I realized that the truly appropriate , low-cost tools that I need
 ed to start a sustainable farm and settlement just did n't exist yet . I needed tools that were robust , modular , highly efficient and optimized , low-cost , made from local and recycled materials that would last a lifetime , not designed for obsolescence . I found that I would have to build them myself . So I did just that . And I tested them . And I found that industrial productivity can be achieved on a small scale . So then I published the 3D designs , schematics , instructional videos and budgets on a wiki . Then contributors from all over the world began showing up , prototyping new machines during dedicated project visits . So far , we have prototyped eight of the 50 machines . And now the project is beginning to grow on its own . We know that open source has succeeded with tools for managing knowledge and creativity . And the same is starting to happen with hardware too . We 're focusing on hardware because it is hardware that can change people 's lives in such tangible mat
 erial ways . If we can lower the barriers to farming , building , manufacturing , then we can unleash just massive amounts of human potential . That 's not only in the developing world . Our tools are being made for the American farmer , builder , entrepreneur , maker . We 've seen lots of excitement from these people , who can now start a construction business , parts manufacturing , organic CSA or just selling power back to the grid . Our goal is a repository of published designs so clear , so complete , that a single burned DVD is effectively a civilization starter kit . I 've planted a hundred trees in a day . I 've pressed 5,000 bricks in one day from the dirt beneath my feet and built a tractor in six days. From what I 've seen , this is only the beginning . If this idea is truly sound , then the implications are significant . A greater distribution of the means of production , environmentally sound supply chains , and a newly-relevant DIY maker culture can hope to transcend a
 rtificial scarcity . We 're exploring the limits of what we all can do to make a better world with open hardware technology . Thank you . ( Applause ) 
\ No newline at end of file


[43/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/stemmer/PStemmer.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/stemmer/PStemmer.java b/opennlp-similarity/src/main/java/opennlp/tools/stemmer/PStemmer.java
new file mode 100644
index 0000000..ef1e4b5
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/stemmer/PStemmer.java
@@ -0,0 +1,521 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.stemmer;
+
+
+	import java.io.IOException;
+	import java.io.InputStream;
+	import java.io.FileInputStream;
+
+	import static org.apache.lucene.util.RamUsageEstimator.NUM_BYTES_CHAR;
+	import org.apache.lucene.util.ArrayUtil;
+
+	/**
+	 *
+	 * Stemmer, implementing the Porter Stemming Algorithm
+	 *
+	 * The Stemmer class transforms a word into its root form.  The input
+	 * word can be provided a character at time (by calling add()), or at once
+	 * by calling one of the various stem(something) methods.
+	 */
+
+	public class PStemmer
+	{
+	  private char[] b;
+	  private int i,    /* offset into b */
+	    j, k, k0;
+	  private boolean dirty = false;
+	  private static final int INITIAL_SIZE = 50;
+
+	  public PStemmer() {
+	    b = new char[INITIAL_SIZE];
+	    i = 0;
+	  }
+
+	  /**
+	   * reset() resets the stemmer so it can stem another word.  If you invoke
+	   * the stemmer by calling add(char) and then stem(), you must call reset()
+	   * before starting another word.
+	   */
+	  public void reset() { i = 0; dirty = false; }
+
+	  /**
+	   * Add a character to the word being stemmed.  When you are finished
+	   * adding characters, you can call stem(void) to process the word.
+	   */
+	  public void add(char ch) {
+	    if (b.length <= i) {
+	      b = ArrayUtil.grow(b, i+1);
+	    }
+	    b[i++] = ch;
+	  }
+
+	  /**
+	   * After a word has been stemmed, it can be retrieved by toString(),
+	   * or a reference to the internal buffer can be retrieved by getResultBuffer
+	   * and getResultLength (which is generally more efficient.)
+	   */
+	  @Override
+	  public String toString() { return new String(b,0,i); }
+
+	  /**
+	   * Returns the length of the word resulting from the stemming process.
+	   */
+	  public int getResultLength() { return i; }
+
+	  /**
+	   * Returns a reference to a character buffer containing the results of
+	   * the stemming process.  You also need to consult getResultLength()
+	   * to determine the length of the result.
+	   */
+	  public char[] getResultBuffer() { return b; }
+
+	  /* cons(i) is true <=> b[i] is a consonant. */
+
+	  private final boolean cons(int i) {
+	    switch (b[i]) {
+	    case 'a': case 'e': case 'i': case 'o': case 'u':
+	      return false;
+	    case 'y':
+	      return (i==k0) ? true : !cons(i-1);
+	    default:
+	      return true;
+	    }
+	  }
+
+	  /* m() measures the number of consonant sequences between k0 and j. if c is
+	     a consonant sequence and v a vowel sequence, and <..> indicates arbitrary
+	     presence,
+
+	          <c><v>       gives 0
+	          <c>vc<v>     gives 1
+	          <c>vcvc<v>   gives 2
+	          <c>vcvcvc<v> gives 3
+	          ....
+	  */
+
+	  private final int m() {
+	    int n = 0;
+	    int i = k0;
+	    while(true) {
+	      if (i > j)
+	        return n;
+	      if (! cons(i))
+	        break;
+	      i++;
+	    }
+	    i++;
+	    while(true) {
+	      while(true) {
+	        if (i > j)
+	          return n;
+	        if (cons(i))
+	          break;
+	        i++;
+	      }
+	      i++;
+	      n++;
+	      while(true) {
+	        if (i > j)
+	          return n;
+	        if (! cons(i))
+	          break;
+	        i++;
+	      }
+	      i++;
+	    }
+	  }
+
+	  /* vowelinstem() is true <=> k0,...j contains a vowel */
+
+	  private final boolean vowelinstem() {
+	    int i;
+	    for (i = k0; i <= j; i++)
+	      if (! cons(i))
+	        return true;
+	    return false;
+	  }
+
+	  /* doublec(j) is true <=> j,(j-1) contain a double consonant. */
+
+	  private final boolean doublec(int j) {
+	    if (j < k0+1)
+	      return false;
+	    if (b[j] != b[j-1])
+	      return false;
+	    return cons(j);
+	  }
+
+	  /* cvc(i) is true <=> i-2,i-1,i has the form consonant - vowel - consonant
+	     and also if the second c is not w,x or y. this is used when trying to
+	     restore an e at the end of a short word. e.g.
+
+	          cav(e), lov(e), hop(e), crim(e), but
+	          snow, box, tray.
+
+	  */
+
+	  private final boolean cvc(int i) {
+	    if (i < k0+2 || !cons(i) || cons(i-1) || !cons(i-2))
+	      return false;
+	    else {
+	      int ch = b[i];
+	      if (ch == 'w' || ch == 'x' || ch == 'y') return false;
+	    }
+	    return true;
+	  }
+
+	  private final boolean ends(String s) {
+	    int l = s.length();
+	    int o = k-l+1;
+	    if (o < k0)
+	      return false;
+	    for (int i = 0; i < l; i++)
+	      if (b[o+i] != s.charAt(i))
+	        return false;
+	    j = k-l;
+	    return true;
+	  }
+
+	  /* setto(s) sets (j+1),...k to the characters in the string s, readjusting
+	     k. */
+
+	  void setto(String s) {
+	    int l = s.length();
+	    int o = j+1;
+	    for (int i = 0; i < l; i++)
+	      b[o+i] = s.charAt(i);
+	    k = j+l;
+	    dirty = true;
+	  }
+
+	  /* r(s) is used further down. */
+
+	  void r(String s) { if (m() > 0) setto(s); }
+
+	  /* step1() gets rid of plurals and -ed or -ing. e.g.
+
+	           caresses  ->  caress
+	           ponies    ->  poni
+	           ties      ->  ti
+	           caress    ->  caress
+	           cats      ->  cat
+
+	           feed      ->  feed
+	           agreed    ->  agree
+	           disabled  ->  disable
+
+	           matting   ->  mat
+	           mating    ->  mate
+	           meeting   ->  meet
+	           milling   ->  mill
+	           messing   ->  mess
+
+	           meetings  ->  meet
+
+	  */
+
+	  private final void step1() {
+	    if (b[k] == 's') {
+	      if (ends("sses")) k -= 2;
+	      else if (ends("ies")) setto("i");
+	      else if (b[k-1] != 's') k--;
+	    }
+	    if (ends("eed")) {
+	      if (m() > 0)
+	        k--;
+	    }
+	    else if ((ends("ed") || ends("ing")) && vowelinstem()) {
+	      k = j;
+	      if (ends("at")) setto("ate");
+	      else if (ends("bl")) setto("ble");
+	      else if (ends("iz")) setto("ize");
+	      else if (doublec(k)) {
+	        int ch = b[k--];
+	        if (ch == 'l' || ch == 's' || ch == 'z')
+	          k++;
+	      }
+	      else if (m() == 1 && cvc(k))
+	        setto("e");
+	    }
+	  }
+
+	  /* step2() turns terminal y to i when there is another vowel in the stem. */
+
+	  private final void step2() {
+	    if (ends("y") && vowelinstem()) {
+	      b[k] = 'i';
+	      dirty = true;
+	    }
+	  }
+
+	  /* step3() maps double suffices to single ones. so -ization ( = -ize plus
+	     -ation) maps to -ize etc. note that the string before the suffix must give
+	     m() > 0. */
+
+	  private final void step3() {
+	    if (k == k0) return; /* For Bug 1 */
+	    switch (b[k-1]) {
+	    case 'a':
+	      if (ends("ational")) { r("ate"); break; }
+	      if (ends("tional")) { r("tion"); break; }
+	      break;
+	    case 'c':
+	      if (ends("enci")) { r("ence"); break; }
+	      if (ends("anci")) { r("ance"); break; }
+	      break;
+	    case 'e':
+	      if (ends("izer")) { r("ize"); break; }
+	      break;
+	    case 'l':
+	      if (ends("bli")) { r("ble"); break; }
+	      if (ends("alli")) { r("al"); break; }
+	      if (ends("entli")) { r("ent"); break; }
+	      if (ends("eli")) { r("e"); break; }
+	      if (ends("ousli")) { r("ous"); break; }
+	      break;
+	    case 'o':
+	      if (ends("ization")) { r("ize"); break; }
+	      if (ends("ation")) { r("ate"); break; }
+	      if (ends("ator")) { r("ate"); break; }
+	      break;
+	    case 's':
+	      if (ends("alism")) { r("al"); break; }
+	      if (ends("iveness")) { r("ive"); break; }
+	      if (ends("fulness")) { r("ful"); break; }
+	      if (ends("ousness")) { r("ous"); break; }
+	      break;
+	    case 't':
+	      if (ends("aliti")) { r("al"); break; }
+	      if (ends("iviti")) { r("ive"); break; }
+	      if (ends("biliti")) { r("ble"); break; }
+	      break;
+	    case 'g':
+	      if (ends("logi")) { r("log"); break; }
+	    }
+	  }
+
+	  /* step4() deals with -ic-, -full, -ness etc. similar strategy to step3. */
+
+	  private final void step4() {
+	    switch (b[k]) {
+	    case 'e':
+	      if (ends("icate")) { r("ic"); break; }
+	      if (ends("ative")) { r(""); break; }
+	      if (ends("alize")) { r("al"); break; }
+	      break;
+	    case 'i':
+	      if (ends("iciti")) { r("ic"); break; }
+	      break;
+	    case 'l':
+	      if (ends("ical")) { r("ic"); break; }
+	      if (ends("ful")) { r(""); break; }
+	      break;
+	    case 's':
+	      if (ends("ness")) { r(""); break; }
+	      break;
+	    }
+	  }
+
+	  /* step5() takes off -ant, -ence etc., in context <c>vcvc<v>. */
+
+	  private final void step5() {
+	    if (k == k0) return; /* for Bug 1 */
+	    switch (b[k-1]) {
+	    case 'a':
+	      if (ends("al")) break;
+	      return;
+	    case 'c':
+	      if (ends("ance")) break;
+	      if (ends("ence")) break;
+	      return;
+	    case 'e':
+	      if (ends("er")) break; return;
+	    case 'i':
+	      if (ends("ic")) break; return;
+	    case 'l':
+	      if (ends("able")) break;
+	      if (ends("ible")) break; return;
+	    case 'n':
+	      if (ends("ant")) break;
+	      if (ends("ement")) break;
+	      if (ends("ment")) break;
+	      /* element etc. not stripped before the m */
+	      if (ends("ent")) break;
+	      return;
+	    case 'o':
+	      if (ends("ion") && j >= 0 && (b[j] == 's' || b[j] == 't')) break;
+	      /* j >= 0 fixes Bug 2 */
+	      if (ends("ou")) break;
+	      return;
+	      /* takes care of -ous */
+	    case 's':
+	      if (ends("ism")) break;
+	      return;
+	    case 't':
+	      if (ends("ate")) break;
+	      if (ends("iti")) break;
+	      return;
+	    case 'u':
+	      if (ends("ous")) break;
+	      return;
+	    case 'v':
+	      if (ends("ive")) break;
+	      return;
+	    case 'z':
+	      if (ends("ize")) break;
+	      return;
+	    default:
+	      return;
+	    }
+	    if (m() > 1)
+	      k = j;
+	  }
+
+	  /* step6() removes a final -e if m() > 1. */
+
+	  private final void step6() {
+	    j = k;
+	    if (b[k] == 'e') {
+	      int a = m();
+	      if (a > 1 || a == 1 && !cvc(k-1))
+	        k--;
+	    }
+	    if (b[k] == 'l' && doublec(k) && m() > 1)
+	      k--;
+	  }
+
+
+	  /**
+	   * Stem a word provided as a String.  Returns the result as a String.
+	   */
+	  public String stem(String s) {
+	    if (stem(s.toCharArray(), s.length()))
+	      return toString();
+	    else
+	      return s;
+	  }
+
+	  /** Stem a word contained in a char[].  Returns true if the stemming process
+	   * resulted in a word different from the input.  You can retrieve the
+	   * result with getResultLength()/getResultBuffer() or toString().
+	   */
+	  public boolean stem(char[] word) {
+	    return stem(word, word.length);
+	  }
+
+	  /** Stem a word contained in a portion of a char[] array.  Returns
+	   * true if the stemming process resulted in a word different from
+	   * the input.  You can retrieve the result with
+	   * getResultLength()/getResultBuffer() or toString().
+	   */
+	  public boolean stem(char[] wordBuffer, int offset, int wordLen) {
+	    reset();
+	    if (b.length < wordLen) {
+	      b = new char[ArrayUtil.oversize(wordLen, NUM_BYTES_CHAR)];
+	    }
+	    System.arraycopy(wordBuffer, offset, b, 0, wordLen);
+	    i = wordLen;
+	    return stem(0);
+	  }
+
+	  /** Stem a word contained in a leading portion of a char[] array.
+	   * Returns true if the stemming process resulted in a word different
+	   * from the input.  You can retrieve the result with
+	   * getResultLength()/getResultBuffer() or toString().
+	   */
+	  public boolean stem(char[] word, int wordLen) {
+	    return stem(word, 0, wordLen);
+	  }
+
+	  /** Stem the word placed into the Stemmer buffer through calls to add().
+	   * Returns true if the stemming process resulted in a word different
+	   * from the input.  You can retrieve the result with
+	   * getResultLength()/getResultBuffer() or toString().
+	   */
+	  public boolean stem() {
+	    return stem(0);
+	  }
+
+	  public boolean stem(int i0) {
+	    k = i - 1;
+	    k0 = i0;
+	    if (k > k0+1) {
+	      step1(); step2(); step3(); step4(); step5(); step6();
+	    }
+	    // Also, a word is considered dirty if we lopped off letters
+	    // Thanks to Ifigenia Vairelles for pointing this out.
+	    if (i != k+1)
+	      dirty = true;
+	    i = k+1;
+	    return dirty;
+	  }
+
+	  /** Test program for demonstrating the Stemmer.  It reads a file and
+	   * stems each word, writing the result to standard out.
+	   * Usage: Stemmer file-name
+	   */
+	  public static void main(String[] args) {
+	    PStemmer s = new PStemmer();
+
+	    for (int i = 0; i < args.length; i++) {
+	      try {
+	        InputStream in = new FileInputStream(args[i]);
+	        byte[] buffer = new byte[1024];
+	        int bufferLen, offset, ch;
+
+	        bufferLen = in.read(buffer);
+	        offset = 0;
+	        s.reset();
+
+	        while(true) {
+	          if (offset < bufferLen)
+	            ch = buffer[offset++];
+	          else {
+	            bufferLen = in.read(buffer);
+	            offset = 0;
+	            if (bufferLen < 0)
+	              ch = -1;
+	            else
+	              ch = buffer[offset++];
+	          }
+
+	          if (Character.isLetter((char) ch)) {
+	            s.add(Character.toLowerCase((char) ch));
+	          }
+	          else {
+	             s.stem();
+	             System.out.print(s.toString());
+	             s.reset();
+	             if (ch < 0)
+	               break;
+	             else {
+	               System.out.print((char) ch);
+	             }
+	           }
+	        }
+
+	        in.close();
+	      }
+	      catch (IOException e) {
+	        System.out.println("error reading " + args[i]);
+	      }
+	    }
+	  }
+	}
+

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/word2vec/W2VDistanceMeasurer.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/word2vec/W2VDistanceMeasurer.java b/opennlp-similarity/src/main/java/opennlp/tools/word2vec/W2VDistanceMeasurer.java
new file mode 100644
index 0000000..f673929
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/word2vec/W2VDistanceMeasurer.java
@@ -0,0 +1,124 @@
+package opennlp.tools.word2vec;
+
+import opennlp.tools.textsimilarity.chunker2matcher.ParserChunker2MatcherProcessor;
+
+import org.deeplearning4j.models.embeddings.WeightLookupTable;
+import org.deeplearning4j.models.embeddings.inmemory.InMemoryLookupTable;
+import org.deeplearning4j.models.embeddings.loader.WordVectorSerializer;
+import org.deeplearning4j.models.word2vec.Word2Vec;
+import org.deeplearning4j.models.word2vec.wordstore.inmemory.InMemoryLookupCache;
+import org.deeplearning4j.text.sentenceiterator.SentenceIterator;
+import org.deeplearning4j.text.sentenceiterator.UimaSentenceIterator;
+import org.deeplearning4j.text.tokenization.tokenizer.preprocessor.CommonPreprocessor;
+import org.deeplearning4j.text.tokenization.tokenizerfactory.DefaultTokenizerFactory;
+import org.deeplearning4j.text.tokenization.tokenizerfactory.TokenizerFactory;
+import org.springframework.core.io.ClassPathResource;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+
+public class W2VDistanceMeasurer {
+	static W2VDistanceMeasurer instance;
+	public Word2Vec vec = null;
+	private String resourceDir = null;
+
+	public synchronized static W2VDistanceMeasurer getInstance() {
+		if (instance == null)
+			instance = new W2VDistanceMeasurer();
+		return instance;
+	}
+
+	public W2VDistanceMeasurer(){
+		if (resourceDir ==null)
+			try {
+				resourceDir = new File( "." ).getCanonicalPath()+"/src/test/resources";
+			} catch (IOException e) {
+				e.printStackTrace();
+				vec = null;
+				return;
+			}
+	
+		String pathToW2V = resourceDir + "/w2v/GoogleNews-vectors-negative300.bin.gz";
+		File gModel = new File(pathToW2V);
+		try {
+			vec = WordVectorSerializer.loadGoogleModel(gModel, true);
+		} catch (IOException e) {
+			System.out.println("Word2vec model is not loaded");
+			vec = null;
+			return;
+		} 
+		
+	} 
+
+	public static void main(String[] args){
+
+		W2VDistanceMeasurer vw2v = W2VDistanceMeasurer.getInstance();
+
+		double value = vw2v.vec.similarity("product", "item");
+		System.out.println(value);
+	}
+
+
+	public static void runCycle() {
+
+		String filePath=null;
+		try {
+			filePath = new ClassPathResource("raw_sentences.txt").getFile().getAbsolutePath();
+		} catch (IOException e1) {
+			// TODO Auto-generated catch block
+			e1.printStackTrace();
+		}
+
+		System.out.println("Load & Vectorize Sentences....");
+		// Strip white space before and after for each line
+		SentenceIterator iter=null;
+		try {
+			iter = UimaSentenceIterator.createWithPath(filePath);
+		} catch (Exception e1) {
+			// TODO Auto-generated catch block
+			e1.printStackTrace();
+		}
+		// Split on white spaces in the line to get words
+		TokenizerFactory t = new DefaultTokenizerFactory();
+		t.setTokenPreProcessor(new CommonPreprocessor());
+
+		InMemoryLookupCache cache = new InMemoryLookupCache();
+		WeightLookupTable table = new InMemoryLookupTable.Builder()
+		.vectorLength(100)
+		.useAdaGrad(false)
+		.cache(cache)
+		.lr(0.025f).build();
+
+		System.out.println("Building model....");
+		Word2Vec vec = new Word2Vec.Builder()
+		.minWordFrequency(5).iterations(1)
+		.layerSize(100).lookupTable(table)
+		.stopWords(new ArrayList<String>())
+		.vocabCache(cache).seed(42)
+		.windowSize(5).iterate(iter).tokenizerFactory(t).build();
+
+		System.out.println("Fitting Word2Vec model....");
+		try {
+			vec.fit();
+		} catch (IOException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		}
+
+		System.out.println("Writing word vectors to text file....");
+		// Write word
+		try {
+			WordVectorSerializer.writeWordVectors(vec, "pathToWriteto.txt");
+		} catch (IOException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		}
+
+		System.out.println("Closest Words:");
+		Collection<String> lst = vec.wordsNearest("day", 10);
+		System.out.println(lst);
+	}
+}
+

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/java/opennlp/tools/fca/FCATest.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/java/opennlp/tools/fca/FCATest.java b/opennlp-similarity/src/test/java/opennlp/tools/fca/FCATest.java
new file mode 100755
index 0000000..531e6ec
--- /dev/null
+++ b/opennlp-similarity/src/test/java/opennlp/tools/fca/FCATest.java
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.fca;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.Arrays;
+import java.util.LinkedHashSet;
+
+import junit.framework.TestCase;
+
+public class FCATest extends TestCase{
+	ConceptLattice cl=null;
+/*
+	public void testConceptLattice() {
+
+
+		try {
+			cl = new ConceptLattice("src/test/resources/fca/sports.cxt",true);
+			cl.printLatticeStats();
+			cl.printLatticeFull();
+			cl.printBinContext();
+
+			FcaWriter wt = new FcaWriter();
+			wt.WriteStatsToCvs("stats.csv", cl, 0);
+		} catch (FileNotFoundException e) {
+			e.printStackTrace();
+		} catch (IOException e) {
+
+			e.printStackTrace();
+		}
+
+		FcaConverter converter = new FcaConverter();
+		int [][] binCon = converter.latticeToContext(cl);
+
+		if (binCon!=null){
+			ConceptLattice new_cl = new ConceptLattice(binCon.length, binCon[0].length, binCon, false);	
+			new_cl.printLatticeStats();
+			new_cl.printLatticeFull();
+			new_cl.printBinContext();
+			FcaWriter wt = new FcaWriter();
+			wt.WriteStatsToCvs("stats.txt", cl, 0);
+			//wt.WriteAsCxt("cl.cxt", cl);
+			wt.WriteAsCxt("cl_new.cxt", new_cl);
+		}
+	}		
+
+	public void testRandom(){
+		RandomNoiseGenerator rng = new RandomNoiseGenerator();
+		try {
+			cl = new ConceptLattice("src/test/resources/fca/sports.cxt",true);
+		} catch (FileNotFoundException e) {
+			e.printStackTrace();
+		} catch (IOException e) {
+			e.printStackTrace();
+		}
+		//int[][] bc = rng.AddObjectsAttributesWithProbability(10, 0.5, cl.binaryContext);
+		int[][] bc = rng.AlterCellsWithProbability(0.2, cl.binaryContext);
+		ConceptLattice new_cl = new ConceptLattice(bc.length, bc[0].length, bc, false);	
+		new_cl.printLatticeStats();
+		new_cl.printLattice();
+	}
+*/
+}
+

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/external_rst/ExternalRSTImporterTest.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/external_rst/ExternalRSTImporterTest.java b/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/external_rst/ExternalRSTImporterTest.java
new file mode 100644
index 0000000..c2b5877
--- /dev/null
+++ b/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/external_rst/ExternalRSTImporterTest.java
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.parse_thicket.external_rst;
+
+
+import java.util.List;
+
+import junit.framework.TestCase;
+import opennlp.tools.parse_thicket.ParseThicket;
+import opennlp.tools.parse_thicket.WordWordInterSentenceRelationArc;
+import opennlp.tools.parse_thicket.matching.Matcher;
+
+public class ExternalRSTImporterTest extends TestCase{
+	
+
+	public void testBuildParseThicketFromTextWithRSTtest(){
+		Matcher m = new Matcher();
+		// We combine our own RST rules with those of Joty 2014 to produce an augmented parse thicket
+		String externalRSTresultFilename = "/external_rst/resInput.txt";
+
+		ParseThicket pt = m.buildParseThicketFromTextWithRST("I explained that I made a deposit, and then wrote a check, which bounced due to a bank error. A customer service representative confirmed that it usually takes a day to process the deposit. "
+				+ "I reminded that I was unfairly charged an overdraft fee amonth ago in a similar situation. "+
+				"  They explained that the overdraft fee was due to insufficient funds as disclosed in my account information. I disagreed with their fee because I made a deposit well in "+
+				" advance and wanted this fee back. They denied responsibility saying that nothing an be done at this point. They also confirmed that I needed to look into the account rules closer.");
+		ExternalRSTImporter imp = new ExternalRSTImporter();
+
+		List<WordWordInterSentenceRelationArc> arcsRST = imp.buildPT2ptPhrases( pt , externalRSTresultFilename);
+		assertTrue(arcsRST .size() > 10);
+
+
+	}
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/pattern_structure/JSMLearnerOnLatticeTest.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/pattern_structure/JSMLearnerOnLatticeTest.java b/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/pattern_structure/JSMLearnerOnLatticeTest.java
new file mode 100644
index 0000000..fd989ba
--- /dev/null
+++ b/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/pattern_structure/JSMLearnerOnLatticeTest.java
@@ -0,0 +1,317 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.parse_thicket.pattern_structure;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.commons.collections.ListUtils;
+
+import junit.framework.TestCase;
+import opennlp.tools.fca.ConceptLattice;
+import opennlp.tools.fca.FcaWriter;
+import opennlp.tools.fca.FormalConcept;
+import opennlp.tools.similarity.apps.BingWebQueryRunner;
+import opennlp.tools.similarity.apps.HitBase;
+import opennlp.tools.similarity.apps.utils.Pair;
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+import opennlp.tools.textsimilarity.ParseTreeChunkListScorer;
+import opennlp.tools.textsimilarity.ParseTreeMatcherDeterministic;
+import opennlp.tools.textsimilarity.chunker2matcher.ParserChunker2MatcherProcessor;
+
+public class JSMLearnerOnLatticeTest extends TestCase{
+	ParserChunker2MatcherProcessor chunk_maker = ParserChunker2MatcherProcessor.getInstance();
+	LinguisticPatternStructure psPos = new LinguisticPatternStructure(0,0), psNeg = new LinguisticPatternStructure(0,0);
+	ParseTreeMatcherDeterministic md = new ParseTreeMatcherDeterministic(); 
+
+	public void testJSMLearner() {
+
+		String text1p = "I rent an office space. This office is for my business. I can deduct office rental expense from my business profit to calculate net income. ";
+		String text2p = "To run my business, I have to rent an office. The net business profit is calculated as follows. Rental expense needs to be subtracted from revenue. ";
+		String text3p = "To store goods for my retail business I rent some space. When I calculate the net income, I take revenue and subtract business expenses such as office rent. ";
+		String text4p = "I rent some space for my business. To calculate my net income, I subtract from revenue my rental business expense.";
+
+
+		String text1n = "I rent out a first floor unit of my house to a travel business. I need to add the rental income to my profit. However, when I repair my house, I can deduct the repair expense from my rental income. ";
+		String text2n = "I receive rental income from my office. I have to claim it as a profit in my tax forms. I need to add my rental income to my profits, but subtract rental expenses such as repair from it. ";
+		String text3n = "I advertised my property as a business rental. Advertisement and repair expenses can be subtracted from the rental income. Remaining rental income needs to be added to my profit and be reported as taxable profit. ";				
+		String text4n = "I showed  my property to a business owner to rent. Expenses on my time spent on advertisement are subtracted from the rental income. My rental profits are added to my taxable income.  ";				
+
+		List<List<ParseTreeChunk>> chunks1p = chunk_maker.formGroupedPhrasesFromChunksForPara(text1p);
+		List<List<ParseTreeChunk>> chunks2p = chunk_maker.formGroupedPhrasesFromChunksForPara(text2p);
+		List<List<ParseTreeChunk>> chunks3p = chunk_maker.formGroupedPhrasesFromChunksForPara(text3p);
+		List<List<ParseTreeChunk>> chunks4p = chunk_maker.formGroupedPhrasesFromChunksForPara(text4p);
+		List<List<ParseTreeChunk>> chunks1n = chunk_maker.formGroupedPhrasesFromChunksForPara(text1n);
+		List<List<ParseTreeChunk>> chunks2n = chunk_maker.formGroupedPhrasesFromChunksForPara(text2n);
+		List<List<ParseTreeChunk>> chunks3n = chunk_maker.formGroupedPhrasesFromChunksForPara(text3n);
+		List<List<ParseTreeChunk>> chunks4n = chunk_maker.formGroupedPhrasesFromChunksForPara(text4n);
+
+
+		LinkedHashSet<Integer> obj = null;
+		obj = new LinkedHashSet<Integer>();
+		obj.add(0);
+		psPos.AddIntent(chunks1p, obj, 0);
+		obj = new LinkedHashSet<Integer>();
+		obj.add(1);
+		psPos.AddIntent(chunks2p, obj, 0);
+		obj = new LinkedHashSet<Integer>();
+		obj.add(2);
+		psPos.AddIntent(chunks3p, obj, 0);
+		obj = new LinkedHashSet<Integer>();
+		obj.add(3);
+		psPos.AddIntent(chunks4p, obj, 0);
+		obj = new LinkedHashSet<Integer>();
+		obj.add(0);
+		psNeg.AddIntent(chunks1n, obj, 0);
+		obj = new LinkedHashSet<Integer>();
+		obj.add(1);
+		psNeg.AddIntent(chunks2n, obj, 0);
+		obj = new LinkedHashSet<Integer>();
+		obj.add(2);
+		psNeg.AddIntent(chunks3n, obj, 0);
+		obj = new LinkedHashSet<Integer>();
+		obj.add(3);
+		psNeg.AddIntent(chunks4n, obj, 0);
+
+		String unknown = "I do not want to rent anything to anyone. I just want to rent a space for myself. I neither calculate deduction of individual or business tax. I subtract my tax from my income";
+		List<List<ParseTreeChunk>> chunksUnknown = chunk_maker.formGroupedPhrasesFromChunksForPara(unknown);
+		List<List<List<ParseTreeChunk>>> posIntersections = new ArrayList<List<List<ParseTreeChunk>>>(), 
+				negIntersections = new ArrayList<List<List<ParseTreeChunk>>>();
+		List<List<ParseTreeChunk>> intersection = null;
+		for(int iConcept = 0; iConcept<psPos.conceptList.size(); iConcept++){
+			if (psPos.conceptList.get(iConcept).intent!=null && psPos.conceptList.get(iConcept).intent.size()>0){
+				intersection = md
+						.matchTwoSentencesGroupedChunksDeterministic(psPos.conceptList.get(iConcept).intent, chunksUnknown);
+				if (reduceList(intersection).size()>0)
+					posIntersections.add(reduceList(intersection));
+			}
+			if (psNeg.conceptList.get(iConcept).intent!=null && psNeg.conceptList.get(iConcept).intent.size()>0){				
+				intersection = md
+						.matchTwoSentencesGroupedChunksDeterministic(psNeg.conceptList.get(iConcept).intent, chunksUnknown);
+				if (reduceList(intersection).size()>0)
+					negIntersections.add(reduceList(intersection));
+			}
+		}
+		
+		Pair<List<List<List<ParseTreeChunk>>>, List<List<List<ParseTreeChunk>>>> pair = 
+		removeInconsistenciesFromPosNegIntersections( posIntersections, 
+			 negIntersections);
+		
+		posIntersections = pair.getFirst();
+		negIntersections = pair.getSecond();
+
+		List<List<List<ParseTreeChunk>>> posIntersectionsUnderNeg = new ArrayList<List<List<ParseTreeChunk>>>(), 
+				negIntersectionsUnderPos = new ArrayList<List<List<ParseTreeChunk>>>();
+
+		for(int iConcept = 0; iConcept<psNeg.conceptList.size(); iConcept++){
+			for(int iConceptJ = 0; iConceptJ<negIntersections.size(); iConceptJ++){
+				intersection = md
+						.matchTwoSentencesGroupedChunksDeterministic(psNeg.conceptList.get(iConcept).intent, negIntersections.get(iConceptJ));
+				if (reduceList(intersection).size()>0)
+					posIntersectionsUnderNeg.add(reduceList(intersection));
+			}
+		}
+
+		for(int iConcept = 0; iConcept<psPos.conceptList.size(); iConcept++){
+			for(int iConceptJ = 0; iConceptJ<posIntersections.size(); iConceptJ++){
+				intersection = md
+						.matchTwoSentencesGroupedChunksDeterministic(psPos.conceptList.get(iConcept).intent, posIntersections.get(iConceptJ));
+				if (reduceList(intersection).size()>0)
+					negIntersectionsUnderPos.add(reduceList(intersection));
+			}
+		}
+
+		List<ParseTreeChunk>posIntersectionsUnderNegLst = flattenParseTreeChunkLst(posIntersectionsUnderNeg);
+		List<ParseTreeChunk>negIntersectionsUnderPosLst=flattenParseTreeChunkLst(negIntersectionsUnderPos);
+
+		posIntersectionsUnderNegLst = subtract(posIntersectionsUnderNegLst, negIntersectionsUnderPosLst);
+		negIntersectionsUnderPosLst= subtract(negIntersectionsUnderPosLst, posIntersectionsUnderNegLst);
+
+		System.out.println("Pos - neg inters = "+posIntersectionsUnderNegLst);
+		System.out.println("Neg - pos inters = "+negIntersectionsUnderPosLst);
+
+	}
+
+	public List<List<ParseTreeChunk>> reduceList(List<List<ParseTreeChunk>> list){
+		float minScore = 1.3f;
+		List<List<ParseTreeChunk>> newList = new ArrayList<List<ParseTreeChunk>>();
+
+
+		ParseTreeChunkListScorer scorer = new ParseTreeChunkListScorer();
+		for(  List<ParseTreeChunk> group: list){
+			List<ParseTreeChunk> newGroup = new ArrayList<ParseTreeChunk>();
+			for(ParseTreeChunk ch: group){
+				if (scorer.getScore(ch) > minScore)
+					newGroup.add(ch);
+			}
+			if (newGroup.size()>0)
+				newList.add(newGroup);
+		} 
+
+		return newList;
+
+	}
+
+	public List<List<ParseTreeChunk>> flattenParseTreeChunkListList(List<List<List<ParseTreeChunk>>> listOfLists){
+		List<List<ParseTreeChunk>> newList = new ArrayList<List<ParseTreeChunk>>();
+
+		for(  List<List<ParseTreeChunk>> member: listOfLists){
+			Set<ParseTreeChunk> newSet= new HashSet<ParseTreeChunk>();
+			for(  List<ParseTreeChunk> group: member){
+				if (group.size()>0)
+					newSet.addAll(group);
+			}
+			newList.add(new ArrayList<ParseTreeChunk>(newSet));
+		}
+
+		return newList;  
+	}
+
+	public List<ParseTreeChunk> flattenParseTreeChunkLst(List<List<List<ParseTreeChunk>>> listOfLists){
+		List<ParseTreeChunk> newList = new ArrayList<ParseTreeChunk>();
+		Set<ParseTreeChunk> newSetAll = new HashSet<ParseTreeChunk>();
+
+
+		for(  List<List<ParseTreeChunk>> member: listOfLists){
+			Set<ParseTreeChunk> newSet= new HashSet<ParseTreeChunk>();
+			for(  List<ParseTreeChunk> group: member){
+				if (group.size()>0)
+					newSet.addAll(group);
+			}
+			newSetAll.addAll(newSet);
+		}
+
+		return removeDuplicates(new ArrayList<ParseTreeChunk>(newSetAll));  
+	}
+
+	public List<ParseTreeChunk> removeDuplicates(List<ParseTreeChunk> dupes){
+		List<Integer> toDelete = new ArrayList<Integer>();
+		for(int i=0; i<dupes.size(); i++)
+			for(int j=i+1; j<dupes.size(); j++){
+				if (dupes.get(i).equals(dupes.get(j))){
+					toDelete.add(j);
+				}
+			}
+		List<ParseTreeChunk> cleaned = new ArrayList<ParseTreeChunk>();
+		for(int i=0; i<dupes.size(); i++){
+			if (!toDelete.contains(i))
+				cleaned.add(dupes.get(i));
+		}
+		return cleaned;
+	}
+
+	public List<ParseTreeChunk> subtract(List<ParseTreeChunk> main, List<ParseTreeChunk> toSubtract){
+		List<Integer> toDelete = new ArrayList<Integer>();
+		for(int i=0; i<main.size(); i++)
+			for(int j=0; j<toSubtract.size(); j++){
+				if (main.get(i).equals(toSubtract.get(j))){
+					toDelete.add(i);
+				}
+			}
+		List<ParseTreeChunk> cleaned = new ArrayList<ParseTreeChunk>();
+		for(int i=0; i<main.size(); i++){
+			if (!toDelete.contains(i))
+				cleaned.add(main.get(i));
+		}
+		return cleaned;
+	}
+	public List<ParseTreeChunk> intesectParseTreeChunkLists(List<ParseTreeChunk> a, List<ParseTreeChunk> b){
+		List<Integer> inters = new ArrayList<Integer>();
+		for(int i=0; i<a.size(); i++)
+			for(int j=0; j<b.size(); j++){
+				if (a.get(i).equals(b.get(j))){
+					inters.add(i);
+				}
+			}
+		List<ParseTreeChunk> cleaned = new ArrayList<ParseTreeChunk>();
+		for(int i=0; i<a.size(); i++){
+			if (inters.contains(i))
+				cleaned.add(a.get(i));
+		}
+		return cleaned;
+	}
+
+	public Pair<List<List<List<ParseTreeChunk>>>, List<List<List<ParseTreeChunk>>>>
+		removeInconsistenciesFromPosNegIntersections(List<List<List<ParseTreeChunk>>> pos, 
+			List<List<List<ParseTreeChunk>>> neg ){
+
+		List<ParseTreeChunk> posIntersectionsFl = flattenParseTreeChunkLst(pos);
+		List<ParseTreeChunk> negIntersectionsFl = flattenParseTreeChunkLst(neg);
+
+		List<ParseTreeChunk> intersParseTreeChunkLists = intesectParseTreeChunkLists(posIntersectionsFl, negIntersectionsFl);
+
+		List<List<List<ParseTreeChunk>>> cleanedFromInconsPos = new ArrayList<List<List<ParseTreeChunk>>>(), 
+				cleanedFromInconsNeg = new ArrayList<List<List<ParseTreeChunk>>>();
+		/*
+		System.out.println("pos = "+ pos);
+		System.out.println("neg = "+ neg);
+		System.out.println("pos flat = "+ posIntersectionsFl);
+		System.out.println("neg flat = "+ negIntersectionsFl);
+		System.out.println("inters = "+  intersParseTreeChunkLists);
+		*/
+
+		for(  List<List<ParseTreeChunk>> member: pos){
+			List<List<ParseTreeChunk>> memberList = new ArrayList<List<ParseTreeChunk>>();
+			for( List<ParseTreeChunk> group: member){
+				List<ParseTreeChunk> newGroup = new ArrayList<ParseTreeChunk>();
+				for(ParseTreeChunk ch: group){
+					boolean bSkip = false;	 
+					for(ParseTreeChunk check: intersParseTreeChunkLists){
+						if (check.equals(ch))
+							bSkip=true;
+					}
+					if (!bSkip)
+						newGroup.add(ch);
+				}
+				if (newGroup.size()>0)
+					memberList.add(newGroup);
+			} 
+			if (memberList.size()>0)
+				cleanedFromInconsPos.add(memberList);
+		}
+		
+		for(  List<List<ParseTreeChunk>> member: neg){
+			List<List<ParseTreeChunk>> memberList = new ArrayList<List<ParseTreeChunk>>();
+			for( List<ParseTreeChunk> group: member){
+				List<ParseTreeChunk> newGroup = new ArrayList<ParseTreeChunk>();
+				for(ParseTreeChunk ch: group){
+					boolean bSkip = false;	 
+					for(ParseTreeChunk check: intersParseTreeChunkLists){
+						if (check.equals(ch))
+							bSkip=true;
+					}
+					if (!bSkip)
+						newGroup.add(ch);
+				}
+				if (newGroup.size()>0)
+					memberList.add(newGroup);
+			} 
+			if (memberList.size()>0)
+				cleanedFromInconsNeg.add(memberList);
+		}
+
+		return  new Pair(cleanedFromInconsPos, cleanedFromInconsNeg);
+
+	}
+
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/pattern_structure/PatternStructureTest.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/pattern_structure/PatternStructureTest.java b/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/pattern_structure/PatternStructureTest.java
new file mode 100644
index 0000000..47324a2
--- /dev/null
+++ b/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/pattern_structure/PatternStructureTest.java
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.parse_thicket.pattern_structure;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.LinkedHashSet;
+import java.util.List;
+
+import junit.framework.TestCase;
+import opennlp.tools.fca.ConceptLattice;
+import opennlp.tools.fca.FcaWriter;
+import opennlp.tools.fca.FormalConcept;
+import opennlp.tools.similarity.apps.BingWebQueryRunner;
+import opennlp.tools.similarity.apps.HitBase;
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+import opennlp.tools.textsimilarity.chunker2matcher.ParserChunker2MatcherProcessor;
+
+public class PatternStructureTest extends TestCase{
+		ParserChunker2MatcherProcessor chunk_maker = ParserChunker2MatcherProcessor.getInstance();
+		LinguisticPatternStructure ps = new LinguisticPatternStructure(0,0);
+		BingWebQueryRunner bqr = new BingWebQueryRunner();
+
+	public void test6texts() {
+
+		String text1 = "I rent an office space. This office is for my business. I can deduct office rental expense from my business profit to calculate net income.";
+		String text2 = "To run my business, I have to rent an office. The net business profit is calculated as follows. Rental expense needs to be subtracted from revenue.";
+		String text3 = "To store goods for my retail business I rent some space. When I calculate the net income, I take revenue and subtract business expenses such as office rent.";
+		String text4 = "I rent out a first floor unit of my house to a travel business. I need to add the rental income to my profit. However, when I repair my house, I can deduct the repair expense from my rental income.";
+		String text5 = "I receive rental income from my office. I have to claim it as a profit in my tax forms. I need to add my rental income to my profits, but subtract rental expenses such as repair from it.";
+		String text6 = "I advertised my property as a business rental. Advertisement and repair expenses can be subtracted from the rental income. Remaining rental income needs to be added to my profit and be reported as taxable profit. ";				
+		
+		List<List<ParseTreeChunk>> chunks1 = chunk_maker.formGroupedPhrasesFromChunksForPara(text1);
+		List<List<ParseTreeChunk>> chunks2 = chunk_maker.formGroupedPhrasesFromChunksForPara(text2);
+		List<List<ParseTreeChunk>> chunks3 = chunk_maker.formGroupedPhrasesFromChunksForPara(text3);
+		List<List<ParseTreeChunk>> chunks4 = chunk_maker.formGroupedPhrasesFromChunksForPara(text4);
+		List<List<ParseTreeChunk>> chunks5 = chunk_maker.formGroupedPhrasesFromChunksForPara(text5);
+		List<List<ParseTreeChunk>> chunks6 = chunk_maker.formGroupedPhrasesFromChunksForPara(text6);
+		//ArrayList<ParseTreeChunk> lst = new ArrayList<ParseTreeChunk>();
+		
+
+		LinkedHashSet<Integer> obj = null;
+		obj = new LinkedHashSet<Integer>();
+		obj.add(0);
+		ps.AddIntent(chunks1, obj, 0);
+		obj = new LinkedHashSet<Integer>();
+		obj.add(1);
+		ps.AddIntent(chunks2, obj, 0);
+		obj = new LinkedHashSet<Integer>();
+		obj.add(2);
+		ps.AddIntent(chunks3, obj, 0);
+		obj = new LinkedHashSet<Integer>();
+		obj.add(3);
+		ps.AddIntent(chunks4, obj, 0);
+		obj = new LinkedHashSet<Integer>();
+		obj.add(4);
+		ps.AddIntent(chunks5, obj, 0);
+		obj = new LinkedHashSet<Integer>();
+		obj.add(5);
+		ps.AddIntent(chunks6, obj, 0);
+
+		ps.logStability();				
+		ps.printLatticeExtended();
+
+		int [][] binaryContext = ps.toContext(3);
+		for (int i = 0; i < binaryContext.length; i++ ){
+			System.out.println(Arrays.toString(binaryContext[i]));
+		}	
+
+		ConceptLattice new_cl = new ConceptLattice(binaryContext.length, binaryContext[0].length, binaryContext,true);	
+		new_cl.printLatticeStats();
+		new_cl.printLatticeFull();
+		assertEquals(new_cl.getLattice().size(), 7);
+		assertEquals(new_cl.getAttributesCount(), 21);
+		assertEquals(new_cl.getObjectCount(), 3);
+	}
+
+				// TEST 2 QUERY NEWS
+	public void testQueryNews(){
+				List<List<ParseTreeChunk>> chunks = null;
+				BingWebQueryRunner bq = new BingWebQueryRunner();
+
+				String q = "";
+//				q = "barack obama";
+// 				q = "lady gaga";
+ 				q = "angela merkel";
+// 				q = "putin";
+				ArrayList <HitBase> hb = (ArrayList<HitBase>) bq.runSearch(q, 10);
+				int cnt = 0;
+				for (HitBase news: hb){
+					LinkedHashSet<Integer> obj = null;
+					obj = new LinkedHashSet<Integer>();
+					obj.add(cnt);
+					chunks = chunk_maker.formGroupedPhrasesFromChunksForPara(news.getAbstractText());
+					System.out.println(chunks);
+					ps.AddIntent(chunks,obj, 0);
+					cnt++;
+				}
+
+				ps.logStability();
+				System.out.println("LATTICE");
+				ps.printLatticeExtended();
+
+				int [][] binaryContext = ps.toContext(cnt);
+				for (int i = 0; i < binaryContext.length; i++ ){
+					System.out.println(Arrays.toString(binaryContext[i]));
+				}	
+
+				ConceptLattice new_cl = new ConceptLattice(binaryContext.length, binaryContext[0].length, binaryContext,true);	
+				new_cl.printLatticeStats();
+				new_cl.printLatticeFull();
+
+				FcaWriter wr = new FcaWriter();
+				wr.WriteAsCxt("res.cxt", new_cl);
+
+				System.out.println("Extent PS "+ps.conceptList.size());
+				//for (int i = 0; i<ps.conceptList.size();i++){
+				//	System.out.println(ps.conceptList.get(i).extent);
+				//}
+				System.out.println("Extent CL "+new_cl.getLattice().size());
+				//for (int i = 0; i<new_cl.getLattice().size();i++){
+				//	System.out.println(new_cl.getLattice().get(i).getExtent());
+				//}
+	}
+	public void testNews(){
+		List<List<ParseTreeChunk>> chunks = null;
+
+				ArrayList <HitBase>  result = (ArrayList<HitBase>) bqr.runSearch("site:http://news.yahoo.com "  + "merkel", 10);
+				System.out.println(" ResultSize  " + result.size());
+				int ind = -1;
+				String text_result = "";
+				for (int i = 0; i < result.size(); i++ ){
+					System.out.println(result.get(i).getAbstractText());
+					ind = result.get(i).getAbstractText().indexOf(") -");
+					if (ind < 0)
+						ind = result.get(i)//.getDescription()
+								.getAbstractText().indexOf(") \ufffd");
+					if (ind > 0)
+						text_result = result.get(i)//.getDescription()
+								.getAbstractText().substring(ind + 3);
+					else 
+						text_result = result.get(i)//.getDescription()
+								.getAbstractText();
+
+					LinkedHashSet<Integer> obj = null;
+					obj = new LinkedHashSet<Integer>();
+					obj.add(i);
+					chunks = chunk_maker.formGroupedPhrasesFromChunksForPara(text_result);
+					ps.AddIntent(chunks,obj, 0);
+				}
+
+				ps.logStability();
+				ps.printLatticeExtended();
+
+				int [][] binaryContext = ps.toContext(result.size());
+
+				ConceptLattice new_cl = new ConceptLattice(binaryContext.length, binaryContext[0].length, binaryContext,true);	
+
+				FcaWriter wt = new FcaWriter();
+				wt.WriteStatsToTxt("merkel_stats.txt", new_cl, 0);
+				wt.WriteStatsToCvs("merkel_stats.csv", new_cl, ps.conceptList.size());
+				wt.WriteAsCxt("merkel_lattice.cxt", new_cl);
+
+				PatternStructureWriter pswt = new PatternStructureWriter();
+				pswt.WriteStatsToTxt("ps_res.txt", ps);
+
+				System.out.println("Extent PS "+ps.conceptList.size());
+				System.out.println("Extent CL "+new_cl.getLattice().size());
+	}
+
+
+
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/pattern_structure/PhraseTest.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/pattern_structure/PhraseTest.java b/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/pattern_structure/PhraseTest.java
new file mode 100755
index 0000000..58246e1
--- /dev/null
+++ b/opennlp-similarity/src/test/java/opennlp/tools/parse_thicket/pattern_structure/PhraseTest.java
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.parse_thicket.pattern_structure;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.LinkedHashSet;
+import java.util.List;
+
+import opennlp.tools.fca.ConceptLattice;
+import opennlp.tools.fca.FcaWriter;
+import opennlp.tools.fca.FormalConcept;
+import opennlp.tools.similarity.apps.BingWebQueryRunner;
+import opennlp.tools.similarity.apps.HitBase;
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+import opennlp.tools.textsimilarity.chunker2matcher.ParserChunker2MatcherProcessor;
+
+public class PhraseTest {
+	
+	
+	public static void main(String []args) {
+		
+/*//TEST 1		
+  				String text1 = "Iran refuses to accept the UN proposal to end its dispute over its work on nuclear weapons."+"UN nuclear watchdog passes a resolution condemning Iran for developing its second uranium enrichment site in secret. " +
+				  "A recent IAEA report presented diagrams that suggested Iran was secretly working on nuclear weapons. " + 
+				  "Iran envoy says its nuclear development is for peaceful purpose, and the material evidence against it has been fabricated by the US. ";
+				String text2 = "However, several experts familiar with the inspections believe that Iraq could also probably have produced a workable device in as little as 6 to 24 months, had they decided to seize foreign-supplied HEU from under safeguards and focus their efforts on a crash program to produce a device in the shortest possible amount of time.";
+				String text3 ="Iraq invested significant resources into uranium enrichment through laser isotope separation (LIS) involving both molecular (MLIS) and atomic vapor (AVLIS) technologies, including a number of activities with respect to laser component manufacture, particularly CO2 lasers and the manufacture of components for use in laser-related experimentation. The Laser Section within the Physics Department of the IAEC at Tuwaitha received an objective in 1981 from the IAEC to work in Laser Isotope Separation. It started in two lines; one which was looking after the molecular and the other the atomic vapor direction.";
+				ParserChunker2MatcherProcessor chunk_maker = ParserChunker2MatcherProcessor.getInstance();
+				List<List<ParseTreeChunk>> chunks1 = chunk_maker.formGroupedPhrasesFromChunksForPara(text1);
+				List<List<ParseTreeChunk>> chunks2 = chunk_maker.formGroupedPhrasesFromChunksForPara(text2);
+				List<List<ParseTreeChunk>> chunks3 = chunk_maker.formGroupedPhrasesFromChunksForPara(text3);
+				ArrayList<ParseTreeChunk> lst = new ArrayList<ParseTreeChunk>();
+				PhrasePatternStructureExtended ps = new PhrasePatternStructureExtended(0,0);
+				LinkedHashSet<Integer> obj = null;
+				obj = new LinkedHashSet<Integer>();
+				obj.add(0);
+				ps.AddIntent(chunks1, obj, 0);
+				obj = new LinkedHashSet<Integer>();
+				obj.add(1);
+				ps.AddIntent(chunks2, obj, 0);
+				obj = new LinkedHashSet<Integer>();
+				obj.add(2);
+				ps.AddIntent(chunks3, obj, 0);
+				ps.logStability();				
+				ps.printLatticeExtended();
+				
+				int [][] binaryContext = ps.toContext(3);
+				for (int i = 0; i < binaryContext.length; i++ ){
+					System.out.println(Arrays.toString(binaryContext[i]));
+				}	
+				
+				ConceptLattice new_cl = new ConceptLattice(binaryContext.length, binaryContext[0].length, binaryContext,true);	
+				new_cl.printLatticeStats();
+				new_cl.printLatticeFull();
+*/
+		
+/*				// TEST 2 QUERY NEWS
+   				ParserChunker2MatcherProcessor chunk_maker = ParserChunker2MatcherProcessor.getInstance();
+				List<List<ParseTreeChunk>> chunks = null;
+				BingWebQueryRunner bq = new BingWebQueryRunner();
+				
+				String q = "";
+//				q = "barack obama";
+// 				q = "lady gaga";
+ 				q = "angela merkel";
+// 				q = "putin";
+				ArrayList <HitBase> hb = (ArrayList<HitBase>) bq.runSearch(q, 10);
+				PhrasePatternStructureExtended ps = new PhrasePatternStructureExtended(0,0);
+				int cnt = 0;
+				for (HitBase news: hb){
+					LinkedHashSet<Integer> obj = null;
+					obj = new LinkedHashSet<Integer>();
+					obj.add(cnt);
+					chunks = chunk_maker.formGroupedPhrasesFromChunksForPara(news.getDescription());
+					System.out.println(chunks);
+					ps.AddIntent(chunks,obj, 0);
+					cnt++;
+				}
+				
+				ps.logStability();
+				System.out.println("LATTICE");
+				ps.printLatticeExtended();
+				
+				int [][] binaryContext = ps.toContext(cnt);
+				for (int i = 0; i < binaryContext.length; i++ ){
+					System.out.println(Arrays.toString(binaryContext[i]));
+				}	
+				
+				ConceptLattice new_cl = new ConceptLattice(binaryContext.length, binaryContext[0].length, binaryContext,true);	
+				new_cl.printLatticeStats();
+				new_cl.printLatticeFull();
+				
+				FcaWriter wr = new FcaWriter();
+				wr.WriteAsCxt("res.cxt", new_cl);
+				
+				System.out.println("Extent PS "+ps.conceptList.size());
+				//for (int i = 0; i<ps.conceptList.size();i++){
+				//	System.out.println(ps.conceptList.get(i).extent);
+				//}
+				System.out.println("Extent CL "+new_cl.getLattice().size());
+				//for (int i = 0; i<new_cl.getLattice().size();i++){
+				//	System.out.println(new_cl.getLattice().get(i).getExtent());
+				//}
+*/				
+				LinguisticPatternStructure ps = new LinguisticPatternStructure(0,0);
+				ParserChunker2MatcherProcessor chunk_maker = ParserChunker2MatcherProcessor.getInstance();
+				List<List<ParseTreeChunk>> chunks = null;		
+				BingWebQueryRunner bqr = new BingWebQueryRunner();
+				ArrayList <HitBase>  result = (ArrayList<HitBase>) bqr.runSearch("site:http://news.yahoo.com "  + "merkel", 10);
+				System.out.println(" ResultSize  " + result.size());
+				int ind = -1;
+				String text_result = "";
+				for (int i = 0; i < result.size(); i++ ){
+					System.out.println(result.get(i).getAbstractText());
+					ind = result.get(i).getAbstractText().indexOf(") -");
+					if (ind < 0)
+						ind = result.get(i)//.getDescription()
+								.getAbstractText().indexOf(") \ufffd");
+					if (ind > 0)
+						text_result = result.get(i)//.getDescription()
+								.getAbstractText().substring(ind + 3);
+					else 
+						text_result = result.get(i)//.getDescription()
+								.getAbstractText();
+					
+					LinkedHashSet<Integer> obj = null;
+					obj = new LinkedHashSet<Integer>();
+					obj.add(i);
+					chunks = chunk_maker.formGroupedPhrasesFromChunksForPara(text_result);
+					ps.AddIntent(chunks,obj, 0);
+				}
+				
+				ps.logStability();
+				ps.printLatticeExtended();
+				
+				int [][] binaryContext = ps.toContext(result.size());
+
+				ConceptLattice new_cl = new ConceptLattice(binaryContext.length, binaryContext[0].length, binaryContext,true);	
+	
+				FcaWriter wt = new FcaWriter();
+				wt.WriteStatsToTxt("merkel_stats.txt", new_cl, 0);
+				wt.WriteStatsToCvs("merkel_stats.csv", new_cl, ps.conceptList.size());
+				wt.WriteAsCxt("merkel_lattice.cxt", new_cl);
+				
+				PatternStructureWriter pswt = new PatternStructureWriter();
+				pswt.WriteStatsToTxt("ps_res.txt", ps);
+				
+				System.out.println("Extent PS "+ps.conceptList.size());
+				System.out.println("Extent CL "+new_cl.getLattice().size());
+
+				
+				}	
+	
+									
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/external_rst/resInput.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/external_rst/resInput.txt b/opennlp-similarity/src/test/resources/external_rst/resInput.txt
new file mode 100644
index 0000000..995b706
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/external_rst/resInput.txt
@@ -0,0 +1,62 @@
+( Root (span 1 21)
+  ( Nucleus (span 1 5) (rel2par span)
+    ( Nucleus (span 1 3) (rel2par Joint)
+      ( Satellite (leaf 1) (rel2par Attribution) (text _!I explained_!) )
+      ( Nucleus (span 2 3) (rel2par span)
+        ( Nucleus (leaf 2) (rel2par span) (text _!that_!) )
+        ( Satellite (leaf 3) (rel2par Attribution) (text _!I made a deposit ,_!) )
+       )
+     )
+    ( Nucleus (span 4 5) (rel2par Joint)
+      ( Nucleus (leaf 4) (rel2par span) (text _!and then wrote a check ,_!) )
+      ( Satellite (leaf 5) (rel2par Elaboration) (text _!which bounced due to a bank error ._!) )
+     )
+  )
+  ( Satellite (span 6 21) (rel2par Summary)
+    ( Nucleus (span 6 8) (rel2par span)
+      ( Satellite (leaf 6) (rel2par Attribution) (text _!A customer service representative confirmed_!) )
+      ( Nucleus (span 7 8) (rel2par span)
+        ( Nucleus (leaf 7) (rel2par span) (text _!that it usually takes a day_!) )
+        ( Satellite (leaf 8) (rel2par Enablement) (text _!to process the deposit ._!) )
+       )
+    )
+    ( Satellite (span 9 21) (rel2par Elaboration)
+      ( Nucleus (span 9 16) (rel2par span)
+        ( Nucleus (span 9 13) (rel2par span)
+          ( Nucleus (span 9 10) (rel2par span)
+            ( Satellite (leaf 9) (rel2par Attribution) (text _!I reminded_!) )
+            ( Nucleus (leaf 10) (rel2par span) (text _!that I was unfairly charged an overdraft fee a month ago in a similar situation ._!) )
+          )
+          ( Satellite (span 11 13) (rel2par Elaboration)
+            ( Satellite (leaf 11) (rel2par Attribution) (text _!They explained_!) )
+            ( Nucleus (span 12 13) (rel2par span)
+              ( Nucleus (leaf 12) (rel2par span) (text _!that the overdraft fee was due to insufficient funds_!) )
+              ( Satellite (leaf 13) (rel2par Comparison) (text _!as disclosed in my account information ._!) )
+             )
+          )
+        )
+        ( Satellite (span 14 16) (rel2par Elaboration)
+          ( Nucleus (leaf 14) (rel2par span) (text _!I disagreed with their fee_!) )
+          ( Satellite (span 15 16) (rel2par Explanation)
+            ( Nucleus (leaf 15) (rel2par Joint) (text _!because I made a deposit well in advance_!) )
+            ( Nucleus (leaf 16) (rel2par Joint) (text _!and wanted this fee back ._!) )
+           )
+        )
+      )
+      ( Satellite (span 17 21) (rel2par Topic-Comment)
+        ( Nucleus (span 17 19) (rel2par Joint)
+          ( Nucleus (leaf 17) (rel2par span) (text _!They denied responsibility_!) )
+          ( Satellite (span 18 19) (rel2par Elaboration)
+            ( Satellite (leaf 18) (rel2par Attribution) (text _!saying_!) )
+            ( Nucleus (leaf 19) (rel2par span) (text _!that nothing can be done at this point ._!) )
+           )
+        )
+        ( Nucleus (span 20 21) (rel2par Joint)
+          ( Satellite (leaf 20) (rel2par Attribution) (text _!They also confirmed_!) )
+          ( Nucleus (leaf 21) (rel2par span) (text _!that I needed to look into the account rules closer ._!) )
+        )
+      )
+    )
+  )
+)
+

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/fca/sports.cxt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/fca/sports.cxt b/opennlp-similarity/src/test/resources/fca/sports.cxt
new file mode 100755
index 0000000..038dc49
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/fca/sports.cxt
@@ -0,0 +1,55 @@
+B
+
+20
+10
+
+Obj 1
+Obj 2
+Obj 3
+Obj 4
+Obj 5
+Obj 6
+Obj 7
+Obj 8
+Obj 9
+Obj 10
+Obj 11
+Obj 12
+Obj 13
+Obj 14
+Obj 15
+Obj 16
+Obj 17
+Obj 18
+Obj 19
+Obj 20
+Attr 1
+Attr 2
+Attr 3
+Attr 4
+Attr 5
+Attr 6
+Attr 7
+Attr 8
+Attr 9
+Attr 10
+X...X....X
+X...X....X
+X...X..XX.
+X.X.X..X.X
+X..X.XX.X.
+X..X.XX.X.
+X..X.XX.X.
+X..X.XX.X.
+.X.X..X.X.
+.X.X....X.
+.X..X....X
+.X.X....X.
+.X..X....X
+.X..X..X.X
+.X..X....X
+..XX...XX.
+..X.X...X.
+..XX.XX.X.
+..X.X...X.
+..XX.....X

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/new_vn.zip
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/new_vn.zip b/opennlp-similarity/src/test/resources/new_vn.zip
new file mode 100644
index 0000000..cf0b9bc
Binary files /dev/null and b/opennlp-similarity/src/test/resources/new_vn.zip differ


[28/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Marx/75MarxMarxK_ComMan_24_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Marx/75MarxMarxK_ComMan_24_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Marx/75MarxMarxK_ComMan_24_EN.txt.txt
new file mode 100644
index 0000000..14d964e
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Marx/75MarxMarxK_ComMan_24_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff II . Proletarians and Communists In what relation do the Communists stand to the proletarians as a whole ? The Communists do not form a separate party opposed to the other working-class parties . They have no interests separate and apart from those of the proletariat as a whole . They do not set up any sectarian principles of their own , by which to shape and mould the proletarian movement . The Communists are distinguished from the other working-class parties by this only : 1. In the national struggles of the proletarians of the different countries , they point out and bring to the front the common interests of the entire proletariat , independently of all nationality . 2. In the various stages of development which the struggle of the working class against the bourgeoisie has to pass through , they always and everywhere represent the interests of the movement as a whole . The Communists , therefore , are on the one hand , practically , the most advanced and resolute section of 
 the working-class parties of every country , that section which pushes forward all others ; on the other hand , theoretically , they have over the great mass of the proletariat the advantage of clearly understanding the line of march , the conditions , and the ultimate general results of the proletarian movement . The immediate aim of the Communists is the same as that of all other proletarian parties : formation of the proletariat into a class , overthrow of the bourgeois supremacy , conquest of political power by the proletariat . The theoretical conclusions of the Communists are in no way based on ideas or principles that have been invented , or discovered , by this or that would-be universal reformer . They merely express , in general terms , actual relations springing from an existing class struggle , from a historical movement going on under our very eyes . The abolition of existing property relations is not at all a distinctive feature of communism . All property relations in
  the past have continually been subject to historical change consequent upon the change in historical conditions . The French Revolution , for example , abolished feudal property in favour of bourgeois property . The distinguishing feature of Communism is not the abolition of property generally , but the abolition of bourgeois property . But modern bourgeois private property is the final and most complete expression of the system of producing and appropriating products , that is based on class antagonisms , on the exploitation of the many by the few . In this sense , the theory of the Communists may be summed up in the single sentence : Abolition of private property . We Communists have been reproached with the desire of abolishing the right of personally acquiring property as the fruit of a man\u2019s own labour , which property is alleged to be the groundwork of all personal freedom , activity and independence . Hard-won , self-acquired , self-earned property ! Do you mean the proper
 ty of petty artisan and of the small peasant , a form of property that preceded the bourgeois form ? There is no need to abolish that ; the development of industry has to a great extent already destroyed it , and is still destroying it daily . Or do you mean the modern bourgeois private property ? But does wage-labour create any property for the labourer ? Not a bit . It creates capital , i. e. , that kind of property which exploits wage-labour , and which cannot increase except upon condition of begetting a new supply of wage-labour for fresh exploitation . Property , in its present form , is based on the antagonism of capital and wage labour . Let us examine both sides of this antagonism . To be a capitalist , is to have not only a purely personal , but a social status in production . Capital is a collective product , and only by the united action of many members , nay , in the last resort , only by the united action of all members of society , can it be set in motion . Capital is
  therefore not only personal ; it is a social power . When , therefore , capital is converted into common property , into the property of all members of society , personal property is not thereby transformed into social property . It is only the social character of the property that is changed . It loses its class character . Let us now take wage-labour . The average price of wage-labour is the minimum wage , i. e. , that quantum of the means of subsistence which is absolutely requisite to keep the labourer in bare existence as a labourer . What , therefore , the wage-labourer appropriates by means of his labour , merely suffices to prolong and reproduce a bare existence . We by no means intend to abolish this personal appropriation of the products of labour , an appropriation that is made for the maintenance and reproduction of human life , and that leaves no surplus wherewith to command the labour of others . All that we want to do away with is the miserable character of this appr
 opriation , under which the labourer lives merely to increase capital , and is allowed to live only in so far as the interest of the ruling class requires it . In bourgeois society , living labour is but a means to increase accumulated labour . In Communist society , accumulated labour is but a means to widen , to enrich , to promote the existence of the labourer . In bourgeois society , therefore , the past dominates the present ; in Communist society , the present dominates the past . In bourgeois society capital is independent and has individuality , while the living person is dependent and has no individuality . And the abolition of this state of things is called by the bourgeois , abolition of individuality and freedom ! And rightly so . The abolition of bourgeois individuality , bourgeois independence , and bourgeois freedom is undoubtedly aimed at . By freedom is meant , under the present bourgeois conditions of production , free trade , free selling and buying . But if selli
 ng and buying disappears , free selling and buying disappears also . This talk about free selling and buying , and all the other \u201c brave words \u201d of our bourgeois about freedom in general , have a meaning , if any , only in contrast with restricted selling and buying , with the fettered traders of the Middle Ages , but have no meaning when opposed to the Communistic abolition of buying and selling , of the bourgeois conditions of production , and of the bourgeoisie itself . You are horrified at our intending to do away with private property . But in your existing society , private property is already done away with for nine-tenths of the population ; its existence for the few is solely due to its non-existence in the hands of those nine-tenths . You reproach us , therefore , with intending to do away with a form of property , the necessary condition for whose existence is the non-existence of any property for the immense majority of society . In one word , you reproach us with in
 tending to do away with your property . Precisely so ; that is just what we intend . From the moment when labour can no longer be converted into capital , money , or rent , into a social power capable of being monopolised , i. e. , from the moment when individual property can no longer be transformed into bourgeois property , into capital , from that moment , you say , individuality vanishes . You must , therefore , confess that by \u201c individual \u201d you mean no other person than the bourgeois , than the middle-class owner of property . This person must , indeed , be swept out of the way , and made impossible . Communism deprives no man of the power to appropriate the products of society ; all that it does is to deprive him of the power to subjugate the labour of others by means of such appropriations . It has been objected that upon the abolition of private property , all work will cease , and universal laziness will overtake us . According to this , bourgeois society ought long ag
 o to have gone to the dogs through sheer idleness ; for those of its members who work , acquire nothing , and those who acquire anything do not work . The whole of this objection is but another expression of the tautology : that there can no longer be any wage-labour when there is no longer any capital . All objections urged against the Communistic mode of producing and appropriating material products , have , in the same way , been urged against the Communistic mode of producing and appropriating intellectual products . Just as , to the bourgeois , the disappearance of class property is the disappearance of production itself , so the disappearance of class culture is to him identical with the disappearance of all culture . That culture , the loss of which he laments , is , for the enormous majority , a mere training to act as a machine . But don\u2019t wrangle with us so long as you apply , to our intended abolition of bourgeois property , the standard of your bourgeois notions of fre
 edom , culture , law , &c . Your very ideas are but the outgrowth of the conditions of your bourgeois production and bourgeois property , just as your jurisprudence is but the will of your class made into a law for all , a will whose essential character and direction are determined by the economical conditions of existence of your class . The selfish misconception that induces you to transform into eternal laws of nature and of reason , the social forms springing from your present mode of production and form of property \u2013 historical relations that rise and disappear in the progress of production \u2013 this misconception you share with every ruling class that has preceded you . What you see clearly in the case of ancient property , what you admit in the case of feudal property , you are of course forbidden to admit in the case of your own bourgeois form of property . Abolition [ Aufhebung ] of the family ! Even the most radical flare up at this infamous proposal of the Communists . O
 n what foundation is the present family , the bourgeois family , based ? On capital , on private gain . In its completely developed form , this family exists only among the bourgeoisie . But this state of things finds its complement in the practical absence of the family among the proletarians , and in public prostitution . The bourgeois family will vanish as a matter of course when its complement vanishes , and both will vanish with the vanishing of capital . Do you charge us with wanting to stop the exploitation of children by their parents ? To this crime we plead guilty . But , you say , we destroy the most hallowed of relations , when we replace home education by social . And your education ! Is not that also social , and determined by the social conditions under which you educate , by the intervention direct or indirect , of society , by means of schools , &c . ? The Communists have not invented the intervention of society in education ; they do but seek to alter the character
  of that intervention , and to rescue education from the influence of the ruling class . The bourgeois clap-trap about the family and education , about the hallowed co-relation of parents and child , becomes all the more disgusting , the more , by the action of Modern Industry , all the family ties among the proletarians are torn asunder , and their children transformed into simple articles of commerce and instruments of labour . But you Communists would introduce community of women , screams the bourgeoisie in chorus . The bourgeois sees his wife a mere instrument of production . He hears that the instruments of production are to be exploited in common , and , naturally , can come to no other conclusion that the lot of being common to all will likewise fall to the women . He has not even a suspicion that the real point aimed at is to do away with the status of women as mere instruments of production . For the rest , nothing is more ridiculous than the virtuous indignation of our bo
 urgeois at the community of women which , they pretend , is to be openly and officially established by the Communists . The Communists have no need to introduce community of women ; it has existed almost from time immemorial . Our bourgeois , not content with having wives and daughters of their proletarians at their disposal , not to speak of common prostitutes , take the greatest pleasure in seducing each other\u2019s wives . Bourgeois marriage is , in reality , a system of wives in common and thus , at the most , what the Communists might possibly be reproached with is that they desire to introduce , in substitution for a hypocritically concealed , an openly legalised community of women . For the rest , it is self-evident that the abolition of the present system of production must bring with it the abolition of the community of women springing from that system , i. e. , of prostitution both public and private . The Communists are further reproached with desiring to abolish countries 
 and nationality . The working men have no country . We cannot take from them what they have not got . Since the proletariat must first of all acquire political supremacy , must rise to be the leading class of the nation , must constitute itself the nation , it is so far , itself national , though not in the bourgeois sense of the word . National differences and antagonism between peoples are daily more and more vanishing , owing to the development of the bourgeoisie , to freedom of commerce , to the world market , to uniformity in the mode of production and in the conditions of life corresponding thereto . The supremacy of the proletariat will cause them to vanish still faster . United action , of the leading civilised countries at least , is one of the first conditions for the emancipation of the proletariat . In proportion as the exploitation of one individual by another will also be put an end to , the exploitation of one nation by another will also be put an end to . In proporti
 on as the antagonism between classes within the nation vanishes , the hostility of one nation to another will come to an end . The charges against Communism made from a religious , a philosophical and , generally , from an ideological standpoint , are not deserving of serious examination . Does it require deep intuition to comprehend that man\u2019s ideas , views , and conception , in one word , man\u2019s consciousness , changes with every change in the conditions of his material existence , in his social relations and in his social life ? What else does the history of ideas prove , than that intellectual production changes its character in proportion as material production is changed ? The ruling ideas of each age have ever been the ideas of its ruling class . When people speak of the ideas that revolutionise society , they do but express that fact that within the old society the elements of a new one have been created , and that the dissolution of the old ideas keeps even pace with the
  dissolution of the old conditions of existence . When the ancient world was in its last throes , the ancient religions were overcome by Christianity . When Christian ideas succumbed in the 18th century to rationalist ideas , feudal society fought its death battle with the then revolutionary bourgeoisie . The ideas of religious liberty and freedom of conscience merely gave expression to the sway of free competition within the domain of knowledge . \u201c Undoubtedly , \u201d it will be said , \u201c religious , moral , philosophical , and juridical ideas have been modified in the course of historical development . But religion , morality , philosophy , political science , and law , constantly survived this change . \u201d \u201c There are , besides , eternal truths , such as Freedom , Justice , etc. , that are common to all states of society . But Communism abolishes eternal truths , it abolishes all religion , and all morality , instead of constituting them on a new basis ; it therefore acts in c
 ontradiction to all past historical experience . \u201d What does this accusation reduce itself to ? The history of all past society has consisted in the development of class antagonisms , antagonisms that assumed different forms at different epochs . But whatever form they may have taken , one fact is common to all past ages , viz . , the exploitation of one part of society by the other . No wonder , then , that the social consciousness of past ages , despite all the multiplicity and variety it displays , moves within certain common forms , or general ideas , which cannot completely vanish except with the total disappearance of class antagonisms . The Communist revolution is the most radical rupture with traditional property relations ; no wonder that its development involved the most radical rupture with traditional ideas . But let us have done with the bourgeois objections to Communism . We have seen above , that the first step in the revolution by the working class is to raise the 
 proletariat to the position of ruling class to win the battle of democracy . The proletariat will use its political supremacy to wrest , by degree , all capital from the bourgeoisie , to centralise all instruments of production in the hands of the State , i. e. , of the proletariat organised as the ruling class ; and to increase the total productive forces as rapidly as possible . Of course , in the beginning , this cannot be effected except by means of despotic inroads on the rights of property , and on the conditions of bourgeois production ; by means of measures , therefore , which appear economically insufficient and untenable , but which , in the course of the movement , outstrip themselves , necessitate further inroads upon the old social order , and are unavoidable as a means of entirely revolutionising the mode of production . These measures will , of course , be different in different countries . Nevertheless , in most advanced countries , the following will be pretty gener
 ally applicable . 1. Abolition of property in land and application of all rents of land to public purposes . 2. A heavy progressive or graduated income tax . 3. Abolition of all rights of inheritance . 4. Confiscation of the property of all emigrants and rebels . 5. Centralisation of credit in the hands of the state , by means of a national bank with State capital and an exclusive monopoly . 6. Centralisation of the means of communication and transport in the hands of the State . 7. Extension of factories and instruments of production owned by the State ; the bringing into cultivation of waste-lands , and the improvement of the soil generally in accordance with a common plan . 8. Equal liability of all to work . Establishment of industrial armies , especially for agriculture . 9. Combination of agriculture with manufacturing industries ; gradual abolition of all the distinction between town and country by a more equable distribution of the populace over the country . 10. Free educat
 ion for all children in public schools . Abolition of children\u2019s factory labour in its present form . Combination of education with industrial production , &c , &c . When , in the course of development , class distinctions have disappeared , and all production has been concentrated in the hands of a vast association of the whole nation , the public power will lose its political character . Political power , properly so called , is merely the organised power of one class for oppressing another . If the proletariat during its contest with the bourgeoisie is compelled , by the force of circumstances , to organise itself as a class , if , by means of a revolution , it makes itself the ruling class , and , as such , sweeps away by force the old conditions of production , then it will , along with these conditions , have swept away the conditions for the existence of class antagonisms and of classes generally , and will thereby have abolished its own supremacy as a class . In place of t
 he old bourgeois society , with its classes and class antagonisms , we shall have an association , in which the free development of each is the condition for the free development of all . III . Socialist and Communist Literature 1. Reactionary Socialism A. Feudal Socialism Owing to their historical position , it became the vocation of the aristocracies of France and England to write pamphlets against modern bourgeois society . In the French Revolution of July 1830 , and in the English reform agitation , these aristocracies again succumbed to the hateful upstart . Thenceforth , a serious political struggle was altogether out of the question . A literary battle alone remained possible . But even in the domain of literature the old cries of the restoration period had become impossible . * In order to arouse sympathy , the aristocracy was obliged to lose sight , apparently , of its own interests , and to formulate their indictment against the bourgeoisie in the interest of the exploited
  working class alone . Thus , the aristocracy took their revenge by singing lampoons on their new masters and whispering in his ears sinister prophesies of coming catastrophe . In this way arose feudal Socialism : half lamentation , half lampoon ; half an echo of the past , half menace of the future ; at times , by its bitter , witty and incisive criticism , striking the bourgeoisie to the very heart\u2019s core ; but always ludicrous in its effect , through total incapacity to comprehend the march of modern history . The aristocracy , in order to rally the people to them , waved the proletarian alms-bag in front for a banner . But the people , so often as it joined them , saw on their hindquarters the old feudal coats of arms , and deserted with loud and irreverent laughter . One section of the French Legitimists and \u201c Young England \u201d exhibited this spectacle . In pointing out that their mode of exploitation was different to that of the bourgeoisie , the feudalists forget that the
 y exploited under circumstances and conditions that were quite different and that are now antiquated . In showing that , under their rule , the modern proletariat never existed , they forget that the modern bourgeoisie is the necessary offspring of their own form of society . For the rest , so little do they conceal the reactionary character of their criticism that their chief accusation against the bourgeois amounts to this , that under the bourgeois r�gime a class is being developed which is destined to cut up root and branch the old order of society . What they upbraid the bourgeoisie with is not so much that it creates a proletariat as that it creates a revolutionary proletariat . In political practice , therefore , they join in all coercive measures against the working class ; and in ordinary life , despite their high-falutin phrases , they stoop to pick up the golden apples dropped from the tree of industry , and to barter truth , love , and honour , for traffic in wool , bee
 troot-sugar , and potato spirits . \u2020 As the parson has ever gone hand in hand with the landlord , so has Clerical Socialism with Feudal Socialism . Nothing is easier than to give Christian asceticism a Socialist tinge . Has not Christianity declaimed against private property , against marriage , against the State ? Has it not preached in the place of these , charity and poverty , celibacy and mortification of the flesh , monastic life and Mother Church ? Christian Socialism is but the holy water with which the priest consecrates the heart-burnings of the aristocrat . B. Petty-Bourgeois Socialism The feudal aristocracy was not the only class that was ruined by the bourgeoisie , not the only class whose conditions of existence pined and perished in the atmosphere of modern bourgeois society . The medieval burgesses and the small peasant proprietors were the precursors of the modern bourgeoisie . In those countries which are but little developed , industrially and commercially , thes
 e two classes still vegetate side by side with the rising bourgeoisie . In countries where modern civilisation has become fully developed , a new class of petty bourgeois has been formed , fluctuating between proletariat and bourgeoisie , and ever renewing itself as a supplementary part of bourgeois society . The individual members of this class , however , are being constantly hurled down into the proletariat by the action of competition , and , as modern industry develops , they even see the moment approaching when they will completely disappear as an independent section of modern society , to be replaced in manufactures , agriculture and commerce , by overlookers , bailiffs and shopmen . In countries like France , where the peasants constitute far more than half of the population , it was natural that writers who sided with the proletariat against the bourgeoisie should use , in their criticism of the bourgeois r�gime , the standard of the peasant and petty bourgeois , and from 
 the standpoint of these intermediate classes , should take up the cudgels for the working class . Thus arose petty-bourgeois Socialism . Sismondi was the head of this school , not only in France but also in England . This school of Socialism dissected with great acuteness the contradictions in the conditions of modern production . It laid bare the hypocritical apologies of economists . It proved , incontrovertibly , the disastrous effects of machinery and division of labour ; the concentration of capital and land in a few hands ; overproduction and crises ; it pointed out the inevitable ruin of the petty bourgeois and peasant , the misery of the proletariat , the anarchy in production , the crying inequalities in the distribution of wealth , the industrial war of extermination between nations , the dissolution of old moral bonds , of the old family relations , of the old nationalities . In its positive aims , however , this form of Socialism aspires either to restoring the old means
  of production and of exchange , and with them the old property relations , and the old society , or to cramping the modern means of production and of exchange within the framework of the old property relations that have been , and were bound to be , exploded by those means . In either case , it is both reactionary and Utopian . Its last words are : corporate guilds for manufacture ; patriarchal relations in agriculture . Ultimately , when stubborn historical facts had dispersed all intoxicating effects of self-deception , this form of Socialism ended in a miserable fit of the blues . C. German or \u201c True \u201d Socialism The Socialist and Communist literature of France , a literature that originated under the pressure of a bourgeoisie in power , and that was the expressions of the struggle against this power , was introduced into Germany at a time when the bourgeoisie , in that country , had just begun its contest with feudal absolutism . German philosophers , would-be philosophers ,
  and beaux esprits ( men of letters ) , eagerly seized on this literature , only forgetting , that when these writings immigrated from France into Germany , French social conditions had not immigrated along with them . In contact with German social conditions , this French literature lost all its immediate practical significance and assumed a purely literary aspect . Thus , to the German philosophers of the Eighteenth Century , the demands of the first French Revolution were nothing more than the demands of \u201c Practical Reason \u201d in general , and the utterance of the will of the revolutionary French bourgeoisie signified , in their eyes , the laws of pure Will , of Will as it was bound to be , of true human Will generally . The work of the German literati consisted solely in bringing the new French ideas into harmony with their ancient philosophical conscience , or rather , in annexing the French ideas without deserting their own philosophic point of view . This annexation took pl
 ace in the same way in which a foreign language is appropriated , namely , by translation . It is well known how the monks wrote silly lives of Catholic Saints over the manuscripts on which the classical works of ancient heathendom had been written . The German literati reversed this process with the profane French literature . They wrote their philosophical nonsense beneath the French original . For instance , beneath the French criticism of the economic functions of money , they wrote \u201c Alienation of Humanity \u201d , and beneath the French criticism of the bourgeois state they wrote \u201c Dethronement of the Category of the General \u201d , and so forth . The introduction of these philosophical phrases at the back of the French historical criticisms , they dubbed \u201c Philosophy of Action \u201d , \u201c True Socialism \u201d , \u201c German Science of Socialism \u201d , \u201c Philosophical Foundation of Socialism \u201d , and so on . The French Socialist and Communist literature was thus completely emascu
 lated . And , since it ceased in the hands of the German to express the struggle of one class with the other , he felt conscious of having overcome \u201c French one-sidedness \u201d and of representing , not true requirements , but the requirements of Truth ; not the interests of the proletariat , but the interests of Human Nature , of Man in general , who belongs to no class , has no reality , who exists only in the misty realm of philosophical fantasy . This German socialism , which took its schoolboy task so seriously and solemnly , and extolled its poor stock-in-trade in such a mountebank fashion , meanwhile gradually lost its pedantic innocence . The fight of the Germans , and especially of the Prussian bourgeoisie , against feudal aristocracy and absolute monarchy , in other words , the liberal movement , became more earnest . By this , the long-wished for opportunity was offered to \u201c True \u201d Socialism of confronting the political movement with the Socialist demands , of hurling
  the traditional anathemas against liberalism , against representative government , against bourgeois competition , bourgeois freedom of the press , bourgeois legislation , bourgeois liberty and equality , and of preaching to the masses that they had nothing to gain , and everything to lose , by this bourgeois movement . German Socialism forgot , in the nick of time , that the French criticism , whose silly echo it was , presupposed the existence of modern bourgeois society , with its corresponding economic conditions of existence , and the political constitution adapted thereto , the very things those attainment was the object of the pending struggle in Germany . To the absolute governments , with their following of parsons , professors , country squires , and officials , it served as a welcome scarecrow against the threatening bourgeoisie . It was a sweet finish , after the bitter pills of flogging and bullets , with which these same governments , just at that time , dosed the Ger
 man working-class risings . While this \u201c True \u201d Socialism thus served the government as a weapon for fighting the German bourgeoisie , it , at the same time , directly represented a reactionary interest , the interest of German Philistines . In Germany , the petty-bourgeois class , a relic of the sixteenth century , and since then constantly cropping up again under the various forms , is the real social basis of the existing state of things . To preserve this class is to preserve the existing state of things in Germany . The industrial and political supremacy of the bourgeoisie threatens it with certain destruction \u2013 on the one hand , from the concentration of capital ; on the other , from the rise of a revolutionary proletariat . \u201c True \u201d Socialism appeared to kill these two birds with one stone . It spread like an epidemic . The robe of speculative cobwebs , embroidered with flowers of rhetoric , steeped in the dew of sickly sentiment , this transcendental robe in which 
 the German Socialists wrapped their sorry \u201c eternal truths \u201d , all skin and bone , served to wonderfully increase the sale of their goods amongst such a public . And on its part German Socialism recognised , more and more , its own calling as the bombastic representative of the petty-bourgeois Philistine . It proclaimed the German nation to be the model nation , and the German petty Philistine to be the typical man . To every villainous meanness of this model man , it gave a hidden , higher , Socialistic interpretation , the exact contrary of its real character . It went to the extreme length of directly opposing the \u201c brutally destructive \u201d tendency of Communism , and of proclaiming its supreme and impartial contempt of all class struggles . With very few exceptions , all the so-called Socialist and Communist publications that now ( 1847 ) circulate in Germany belong to the domain of this foul and enervating literature . * 2. Conservative or Bourgeois Socialism A part of th
 e bourgeoisie is desirous of redressing social grievances in order to secure the continued existence of bourgeois society . To this section belong economists , philanthropists , humanitarians , improvers of the condition of the working class , organisers of charity , members of societies for the prevention of cruelty to animals , temperance fanatics , hole-and-corner reformers of every imaginable kind . This form of socialism has , moreover , been worked out into complete systems . We may cite Proudhon\u2019s Philosophie de la Mis�re as an example of this form . The Socialistic bourgeois want all the advantages of modern social conditions without the struggles and dangers necessarily resulting therefrom . They desire the existing state of society , minus its revolutionary and disintegrating elements . They wish for a bourgeoisie without a proletariat . The bourgeoisie naturally conceives the world in which it is supreme to be the best ; and bourgeois Socialism develops this comfortabl
 e conception into various more or less complete systems . In requiring the proletariat to carry out such a system , and thereby to march straightway into the social New Jerusalem , it but requires in reality , that the proletariat should remain within the bounds of existing society , but should cast away all its hateful ideas concerning the bourgeoisie . A second , and more practical , but less systematic , form of this Socialism sought to depreciate every revolutionary movement in the eyes of the working class by showing that no mere political reform , but only a change in the material conditions of existence , in economical relations , could be of any advantage to them . By changes in the material conditions of existence , this form of Socialism , however , by no means understands abolition of the bourgeois relations of production , an abolition that can be affected only by a revolution , but administrative reforms , based on the continued existence of these relations ; reforms , 
 therefore , that in no respect affect the relations between capital and labour , but , at the best , lessen the cost , and simplify the administrative work , of bourgeois government . Bourgeois Socialism attains adequate expression when , and only when , it becomes a mere figure of speech . Free trade : for the benefit of the working class . Protective duties : for the benefit of the working class . Prison Reform : for the benefit of the working class . This is the last word and the only seriously meant word of bourgeois socialism . It is summed up in the phrase : the bourgeois is a bourgeois \u2013 for the benefit of the working class . 3. Critical-Utopian Socialism and Communism We do not here refer to that literature which , in every great modern revolution , has always given voice to the demands of the proletariat , such as the writings of Babeuf and others . The first direct attempts of the proletariat to attain its own ends , made in times of universal excitement , when feudal so
 ciety was being overthrown , necessarily failed , owing to the then undeveloped state of the proletariat , as well as to the absence of the economic conditions for its emancipation , conditions that had yet to be produced , and could be produced by the impending bourgeois epoch alone . The revolutionary literature that accompanied these first movements of the proletariat had necessarily a reactionary character . It inculcated universal asceticism and social levelling in its crudest form . The Socialist and Communist systems , properly so called , those of Saint-Simon , Fourier , Owen , and others , spring into existence in the early undeveloped period , described above , of the struggle between proletariat and bourgeoisie ( see Section I. Bourgeois and Proletarians ) . The founders of these systems see , indeed , the class antagonisms , as well as the action of the decomposing elements in the prevailing form of society . But the proletariat , as yet in its infancy , offers to them t
 he spectacle of a class without any historical initiative or any independent political movement . Since the development of class antagonism keeps even pace with the development of industry , the economic situation , as they find it , does not as yet offer to them the material conditions for the emancipation of the proletariat . They therefore search after a new social science , after new social laws , that are to create these conditions . Historical action is to yield to their personal inventive action ; historically created conditions of emancipation to fantastic ones ; and the gradual , spontaneous class organisation of the proletariat to an organisation of society especially contrived by these inventors . Future history resolves itself , in their eyes , into the propaganda and the practical carrying out of their social plans . In the formation of their plans , they are conscious of caring chiefly for the interests of the working class , as being the most suffering class . Only fr
 om the point of view of being the most suffering class does the proletariat exist for them . The undeveloped state of the class struggle , as well as their own surroundings , causes Socialists of this kind to consider themselves far superior to all class antagonisms . They want to improve the condition of every member of society , even that of the most favoured . Hence , they habitually appeal to society at large , without the distinction of class ; nay , by preference , to the ruling class . For how can people , when once they understand their system , fail to see in it the best possible plan of the best possible state of society ? Hence , they reject all political , and especially all revolutionary action ; they wish to attain their ends by peaceful means , necessarily doomed to failure , and by the force of example , to pave the way for the new social Gospel . Such fantastic pictures of future society , painted at a time when the proletariat is still in a very undeveloped state a
 nd has but a fantastic conception of its own position , correspond with the first instinctive yearnings of that class for a general reconstruction of society . But these Socialist and Communist publications contain also a critical element . They attack every principle of existing society . Hence , they are full of the most valuable materials for the enlightenment of the working class . The practical measures proposed in them \u2013 such as the abolition of the distinction between town and country , of the family , of the carrying on of industries for the account of private individuals , and of the wage system , the proclamation of social harmony , the conversion of the function of the state into a more superintendence of production \u2013 all these proposals point solely to the disappearance of class antagonisms which were , at that time , only just cropping up , and which , in these publications , are recognised in their earliest indistinct and undefined forms only . These proposals , th
 erefore , are of a purely Utopian character . The significance of Critical-Utopian Socialism and Communism bears an inverse relation to historical development . In proportion as the modern class struggle develops and takes definite shape , this fantastic standing apart from the contest , these fantastic attacks on it , lose all practical value and all theoretical justification . Therefore , although the originators of these systems were , in many respects , revolutionary , their disciples have , in every case , formed mere reactionary sects . They hold fast by the original views of their masters , in opposition to the progressive historical development of the proletariat . They , therefore , endeavour , and that consistently , to deaden the class struggle and to reconcile the class antagonisms . They still dream of experimental realisation of their social Utopias , of founding isolated \u201c phalansteres \u201d , of establishing \u201c Home Colonies \u201d , or setting up a \u201c Little Icaria
 * \u2013 duodecimo editions of the New Jerusalem \u2013 and to realise all these castles in the air , they are compelled to appeal to the feelings and purses of the bourgeois . By degrees , they sink into the category of the reactionary [ or ] conservative Socialists depicted above , differing from these only by more systematic pedantry , and by their fanatical and superstitious belief in the miraculous effects of their social science . They , therefore , violently oppose all political action on the part of the working class ; such action , according to them , can only result from blind unbelief in the new Gospel . The Owenites in England , and the Fourierists in France , respectively , oppose the Chartists and the R�formistes . IV . Position of the Communists in Relation to the Various Existing Opposition Parties Section II has made clear the relations of the Communists to the existing working-class parties , such as the Chartists in England and the Agrarian Reformers in America . The 
 Communists fight for the attainment of the immediate aims , for the enforcement of the momentary interests of the working class ; but in the movement of the present , they also represent and take care of the future of that movement . In France , the Communists ally with the Social-Democrats* against the conservative and radical bourgeoisie , reserving , however , the right to take up a critical position in regard to phases and illusions traditionally handed down from the great Revolution . In Switzerland , they support the Radicals , without losing sight of the fact that this party consists of antagonistic elements , partly of Democratic Socialists , in the French sense , partly of radical bourgeois . In Poland , they support the party that insists on an agrarian revolution as the prime condition for national emancipation , that party which fomented the insurrection of Cracow in 1846. In Germany , they fight with the bourgeoisie whenever it acts in a revolutionary way , against the 
 absolute monarchy , the feudal squirearchy , and the petty bourgeoisie . But they never cease , for a single instant , to instil into the working class the clearest possible recognition of the hostile antagonism between bourgeoisie and proletariat , in order that the German workers may straightway use , as so many weapons against the bourgeoisie , the social and political conditions that the bourgeoisie must necessarily introduce along with its supremacy , and in order that , after the fall of the reactionary classes in Germany , the fight against the bourgeoisie itself may immediately begin . The Communists turn their attention chiefly to Germany , because that country is on the eve of a bourgeois revolution that is bound to be carried out under more advanced conditions of European civilisation and with a much more developed proletariat than that of England was in the seventeenth , and France in the eighteenth century , and because the bourgeois revolution in Germany will be but th
 e prelude to an immediately following proletarian revolution . In short , the Communists everywhere support every revolutionary movement against the existing social and political order of things . In all these movements , they bring to the front , as the leading question in each , the property question , no matter what its degree of development at the time . Finally , they labour everywhere for the union and agreement of the democratic parties of all countries . The Communists disdain to conceal their views and aims . They openly declare that their ends can be attained only by the forcible overthrow of all existing social conditions . Let the ruling classes tremble at a Communistic revolution . The proletarians have nothing to lose but their chains . They have a world to win . Working Men of All Countries , Unite ! 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Marx/76MarxMarxK_Feuerbach_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Marx/76MarxMarxK_Feuerbach_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Marx/76MarxMarxK_Feuerbach_EN.txt.txt
new file mode 100644
index 0000000..51999a5
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Marx/76MarxMarxK_Feuerbach_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff -------------------------------------------------------------------------------- I The chief defect of all hitherto existing materialism \u2013 that of Feuerbach included \u2013 is that the thing , reality , sensuousness , is conceived only in the form of the object or of contemplation , but not as sensuous human activity , practice , not subjectively . Hence , in contradistinction to materialism , the active side was developed abstractly by idealism \u2013 which , of course , does not know real , sensuous activity as such . Feuerbach wants sensuous objects , really distinct from the thought objects , but he does not conceive human activity itself as objective activity . Hence , in The Essence of Christianity , he regards the theoretical attitude as the only genuinely human attitude , while practice is conceived and fixed only in its dirty-judaical manifestation . Hence he does not grasp the significance of \u201c revolutionary \u201d , of \u201c practical-critical \u201d , activity . II The questio
 n whether objective truth can be attributed to human thinking is not a question of theory but is a practical question . Man must prove the truth \u2014 i. e. the reality and power , the this-sidedness of his thinking in practice . The dispute over the reality or non-reality of thinking that is isolated from practice is a purely scholastic question . III The materialist doctrine concerning the changing of circumstances and upbringing forgets that circumstances are changed by men and that it is essential to educate the educator himself . This doctrine must , therefore , divide society into two parts , one of which is superior to society . The coincidence of the changing of circumstances and of human activity or self-changing can be conceived and rationally understood only as revolutionary practice . IV Feuerbach starts out from the fact of religious self-alienation , of the duplication of the world into a religious world and a secular one . His work consists in resolving the religious wo
 rld into its secular basis . But that the secular basis detaches itself from itself and establishes itself as an independent realm in the clouds can only be explained by the cleavages and self-contradictions within this secular basis . The latter must , therefore , in itself be both understood in its contradiction and revolutionized in practice . Thus , for instance , after the earthly family is discovered to be the secret of the holy family , the former must then itself be destroyed in theory and in practice . V Feuerbach , not satisfied with abstract thinking , wants contemplation ; but he does not conceive sensuousness as practical , human-sensuous activity . VI Feuerbach resolves the religious essence into the human essence . But the human essence is no abstraction inherent in each single individual . In its reality it is the ensemble of the social relations . Feuerbach , who does not enter upon a criticism of this real essence , is consequently compelled : 1. To abstract from t
 he historical process and to fix the religious sentiment as something by itself and to presuppose an abstract \u2013 isolated \u2013 human individual . 2. Essence , therefore , can be comprehended only as \u201c genus \u201d , as an internal , dumb generality which naturally unites the many individuals . VII Feuerbach , consequently , does not see that the \u201c religious sentiment \u201d is itself a social product , and that the abstract individual whom he analyses belongs to a particular form of society . VIII All social life is essentially practical . All mysteries which lead theory to mysticism find their rational solution in human practice and in the comprehension of this practice . IX The highest point reached by contemplative materialism , that is , materialism which does not comprehend sensuousness as practical activity , is contemplation of single individuals and of civil society . X The standpoint of the old materialism is civil society ; the standpoint of the new is human society , or soc
 ial humanity . XI The philosophers have only interpreted the world , in various ways ; the point is to change it . -------------------------------------------------------------------------------- 
\ No newline at end of file


[29/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/73FictZolaE_Germinal_VII2_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/73FictZolaE_Germinal_VII2_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/73FictZolaE_Germinal_VII2_EN.txt.txt
new file mode 100644
index 0000000..71f43ac
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/73FictZolaE_Germinal_VII2_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Part 7 Chapter 2 ON Sunday �tienne escaped from the settlement at nightfall . A very clear sky , sprinkled with stars , lit up the earth with the blue haze of twilight . He went down towards the canal , and followed the bank slowly , in the direction of Marchiennes . It was his favourite walk , a grass-covered path two leagues long , passing straight beside this geometrical water-way , which unrolled itself like an endless ingot of molten silver . He never met any one there . But on this day he was vexed to see a man come up to him . Beneath the pale starlight , the two solitary walkers only recognized each other when they were face to face . " What ! is it you ? " said �tienne . Souvarine nodded his head without replying . For a moment they remained motionless , then side by side they set out towards Marchiennes . Each of them seemed to be continuing his own reflections , as though they were far away from each other . " Have you seen in the paper about Pluchart 's success at 
 Paris ? " asked �tienne , at length . " After that meeting at Belleville , they waited for him on the pavement , and gave him an ovation . Oh ! he 's afloat now , in spite of his sore throat . He can do what he likes in the future . " The engine-man shrugged his shoulders . He felt contempt for fine talkers , fellows who go into politics as one goes to the bar , to get an income out of phrases . �tienne was now studying Darwin . He had read fragments , summarized and popularized in a five-sou volume ; and out of this ill-understood reading he had gained for himself a revolutionary idea of the struggle for existence , the lean eating the fat , the strong people devouring the pallid middle class . But Souvarine furiously attacked the stupidity of the Socialists who accept Darwin , that apostle of scientific inequality , whose famous selection was only good for aristocratic philosophers . His mate persisted , however , wishing to reason out the matter , and expressing his doubts by a
 n hypothesis : supposing the old society were no longer to exist , swept away to the crumbs ; well , was it not to be feared that the new world would grow up again , slowly spoilt by the same injustices , some sick and others flourishing , some more skilful and intelligent , fattening on everything , and others imbecile and lazy . becoming slaves again ? But before this vision of eternal wretchedness , the engine-man shouted out fiercely that if justice was not possible with man , then man must disappear . For every rotten society there must be a massacre , until the last creature was exterminated . And there was silence again . For a long time , with sunken head , Souvarine walked over the short grass , so absorbed that he kept to the extreme edge , by the water , with the quiet certainty of a sleep-walker on a roof . Then he shuddered causelessly , as though he had stumbled against a shadow . His eyes lifted and his face was very pale ; he said softly to his companion : " Did I ev
 er tell you how she died ? " " Whom do you mean ? " " My woman , over there , in Russia . " �tienne made a vague gesture , astonished at the tremor in his voice and at the sudden desire for confidence in this lad , who was usually so impassive in his stoical detachment from others and from himself . He only knew that the woman was his mistress , and that she had been hanged at Moscow . " The affair had n't gone off , " Souvarine said , with eyes still vacantly following the white stream of the canal between the bluish colonnades of tall trees . " We had been a fortnight at the bottom of a hole undermining the railway , and it was not the imperial train that was blown up , it was a passenger train . Then they arrested Annutchka . She brought us bread every evening , disguised as a peasant woman . She lit the fuse , too , because a man might have attracted attention . I followed the trial , hidden in the crowd , for six days. " His voice became thick , and he coughed as though he wer
 e choking . " Twice I wanted to cry out , and to rush over the people 's heads to join her . But what was the good ? One man less would be one soldier less ; and I could see that she was telling me not to come , when her large eyes met mine . " He coughed again . " On the last day in the square I was there . It was raining ; they stupidly lost their heads , put out by the falling rain . It took twenty minutes to hang the other four ; the cord broke , they could not finish the fourth . Annutchka was standing up waiting . She could not see me , she was looking for me in the crowd . I got on to a post and she saw me , and our eyes never turned from each other . When she was dead she was still looking at me . I waved my hat ; I came away . " There was silence again . The white road of the canal unrolled to the far distance , and they both walked with the same quiet step as though each had fallen back into his isolation . At the horizon , the pale water seemed to open the sky with a litt
 le hole of light . " It was our punishment , " Souvarine went on roughly . " We were guilty to love each other . Yes , it is well that she is dead ; heroes will be born from her blood , and I no longer have any cowardice at my heart . Ah ! nothing , neither parents , nor wife , nor friend ! Nothing to make my hand tremble on the day when I must take others ' lives or give up my own . " �tienne had stopped , shuddering in the cool night . He discussed no more , he simply said : " We have gone far ; shall we go back ? " They went back towards the Voreux slowly , and he added , after a few paces : " Have you seen the new placards ? " The Company had that morning put up some more large yellow posters . They were clearer and more conciliatory , and the Company undertook to take back the certificates of those miners who went down on the following day . Everything would be forgotten , and pardon was offered even to those who were most implicated . " Yes , I 've seen , " replied the engine
 -man . " Well , what do you think of it ? " " I think that it 's all up . The flock will go down again . You are all too cowardly . " �tienne feverishly excused his mates : a man may be brave , a mob which is dying of hunger has no strength . Step by step they were returning to the Voreux ; and before the black mass of the pit he continued swearing that he , at least , would never go down ; but he could forgive those who did . Then , as the rumour ran that the carpenters had not had time to repair the tubbing , he asked for information . Was it true ? Had the weight of the soil against the timber which formed the internal skirt of scaffolding to the shaft so pushed it in that the winding-cages rubbed as they went down for a length of over fifty metres ? Souvarine , who once more became uncommunicative , replied briefly . He had been working the day before , and the cage did , in fact , jar ; the engine-men had even had to double the speed to pass that spot . But all the bosses rece
 ived any observations with the same irritating remark : it was coal they wanted ; that could be repaired later on . " You see that is smashing up ! " �tienne murmured . " It will be a fine time ! " With eyes vaguely fixed on the pit in the shadow , Souvarine quietly concluded : " If it does smash up , the mates will know it , since you advise them to go down again . " Nine o'clock struck at the Montsou steeple ; and his companion having said that he was going to bed , he added , without putting out his hand : " Well , good-bye . I 'm going away . " " What ! you 're going away ? " " Yes , I 've asked for my certificate back . I 'm going elsewhere . " �tienne , stupefied and affected , looked at him . After walking for two hours he said that to him ! And in so calm a voice , while the mere announcement of this sudden separation made his own heart ache . They had got to know each other , they had toiled together ; that always makes one sad , the idea of not seeing a person again . " 
 You 're going away ! And where do you go ? " " Over there--I do n't know at all . " " But I shall see you again ? " " No , I think not . " They were silent and remained for a moment facing each other without finding anything to say . " Then good-bye . " " Good-bye . " While �tienne ascended toward the settlement , Souvarine turned and again went along the canal bank ; and there , now alone , he continued to walk , with sunken head , so lost in the darkness that he seemed merely a moving shadow of the night . Now and then he stopped , he counted the hours that struck afar . When he heard midnight strike he left the bank and turned towards the Voreux . At that time the pit was empty , and he only met a sleepy-eyed captain . It was not until two o'clock that they would begin to get up steam to resume work . First he went to take from a cupboard a jacket which he pretended to have forgotten . Various tools--a drill armed with its screw , a small but very strong saw , a hammer , and a c
 hisel--were rolled up in this jacket . Then he left . But instead of going out through the shed he passed through the narrow corridor which led to the ladder passage . With his jacket under his arm he quietly went down without a lamp , measuring the depth by counting the ladders . He knew that the cage jarred at three hundred and seventy-four metres against the fifth row of the lower tubbing . When he had counted fifty-four ladders he put out his hand and was able to feel the swelling of the planking . It was there . Then , with the skill and coolness of a good workman who has been reflecting over his task for a long time , he set to work . He began by sawing a panel in the brattice so as to communicate with the winding-shaft . With the help of matches , quickly lighted and blown out , he was then able to ascertain the condition of the tubbing and of the recent repairs . Between Calais and Valenciennes the sinking of mine shafts was surrounded by immense difficulties on account of t
 he masses of subterranean water in great sheets at the level of the lowest valleys . Only the construction of tubbings , frameworks jointed like the stays of a barrel , could keep Out the springs which flow in and isolate the shafts in the midst of the lakes , which with deep obscure waves beat against the walls . It had been necessary in sinking the Voreux to establish two tubbings : that of the upper level , in the shifting sands and white clays bordering the chalky stratum , and fissured in every part , swollen with water like a sponge ; then that of the lower level , immediately above the coal stratum , in a yellow sand as fine as flour , flowing with liquid fluidity ; it was here that the Torrent was to be found , that subterranean sea so dreaded in the coal pits of the Nord , a sea with its storms and its shipwrecks , an unknown and unfathomable sea , rolling its dark floods more than three hundred metres beneath the daylight . Usually the tubbings resisted the enormous pressu
 re ; the only thing to be dreaded was the piling up of the neighbouring soil , shaken by the constant movement of the old galleries which were filling up . In this descent of the rocks lines of fracture were sometimes produced which slowly extended as far as the scaffolding , at last perforating it and pushing it into the shaft ; and there was the great danger of a landslip and a flood filling the pit with an avalanche of earth and a deluge of springs . Souvarine , sitting astride in the opening he had made , discovered a very serious defect in the fifth row of tubbing . The wood was bellied out from the framework ; several planks had even come out of their shoulder-pieces . Abundant filtrations , pichoux the miners call them , were jetting out of the joints through the tarred oakum with which they were caulked . The carpenters , pressed for time , had been content to place iron squares at the angles , so carelessly that not all the screws were put in . A considerable movement was e
 vidently going on behind in the sand of the Torrent . Then with his wimble he unscrewed the squares so that another push would tear them all off . It was a foolhardy task , during which he frequently only just escaped from falling headlong down the hundred and eighty metres which separated him from the bottom . He had been obliged to seize the oak guides , the joists along which the cages slid ; and suspended over the void he traversed the length of the cross-beams with which they were joined from point to point , slipping along , sitting down , turning over , simply buttressing himself on an elbow or a knee , with tranquil contempt of death . A breath would have sent him over , and three times he caught himself up without a shudder . First he felt with his hand and then worked , only lighting a match when he lost himself in the midst of these slimy beams . After loosening the screws he attacked the wood itself , and the peril became still greater . He had sought for the key , the p
 iece which held the others ; he attacked it furiously , making holes in it , sawing it , thinning it so that it lost its resistance ; while through the holes and the cracks the water which escaped in small jets blinded him and soaked him in icy rain . Two matches were extinguished . They all be-came damp and then there was night , the bottomless depth of darkness . From this moment he was seized by rage . The breath of the invisible intoxicated him , the black horror of this rain-beaten hole urged him to mad destruction . He wreaked his fury at random against the tubbing , striking where he could with his wimble , with his saw , seized by the desire to bring the whole thing at once down on his head . He brought as much ferocity to the task as though he had been digging a knife into the skin of some execrated living creature . He would kill the Voreux at last , that evil beast with ever-open jaws which had swallowed so much human flesh ! The bite of his tools could be heard , his spi
 ne lengthened , he crawled , climbed down , then up again , holding on by a miracle , in continual movement , the flight of a nocturnal bird amid the scaffolding of a belfry . But he grew calm , dissatisfied with himself . Why could not things be done coolly ? Without haste he took breath , and then went back into the ladder passage , stopping up the hole by replacing the panel which he had sawn . That was enough ; he did not wish to raise the alarm by excessive damage which would have been repaired immediately . The beast was wounded in the belly ; we should see if it was still alive at night . And he had left his mark ; the frightened world would know that the beast had not died a natural death . He took his time in methodically rolling up his tools in his jacket , and slowly climbed up the ladders . Then , when he had emerged from the pit without being seen , it did not even occur to him to go and change his clothes . Three o'clock struck . He remained standing on the road waitin
 g . At the same hour �tienne , who was not asleep , was disturbed by a slight sound in the thick night of the room . He distinguished the low breath of the children , and the snoring of Bonnemort and Maheude ; while Jeanlin near him was breathing with a prolonged flute-like whistle . No doubt he had dreamed , and he was turning back when the noise began again . It was the creaking of a palliasse , the stifled effort of someone who is getting up . Then he imagined that Catherine must be ill . " I say , is it you ? What is the matter ? " he asked in a low voice . No one replied , and the snoring of the others continued . For five minutes nothing stirred . Then there was fresh creaking . Feeling certain this time that he was not mistaken , he crossed the room , putting his hands out into the darkness to feel the opposite bed . He was surprised to find the young girl sitting up , holding in her breath , awake and on the watch . " Well ! why do n't you reply ? What are you doing , then 
 ? " At last she said : " I 'm getting up . " " Getting up at this hour ? " " Yes , I 'm going back to work at the pit . " �tienne felt deeply moved , and sat down on the edge of the palliasse , while Catherine explained her reasons to him . She suffered too much by living thus in idleness , feeling continual looks of reproach weighing on her ; she would rather run the risk of being knocked about down there by Chaval . And if her mother refused to take her money when she brought it , well ! she was big enough to act for herself and make her own soup . " Go away ; I want to dress . And do n't say anything , will you , if you want to be kind ? " But he remained near her ; he had put his arms round her waist in a caress of grief and pity . Pressed one against the other in their shirts , they could feel the warmth of each other 's naked flesh , at the edge of this bed , still moist with the night 's sleep . She had at first tried to free herself ; then she began to cry quietly , in her 
 turn taking him by the neck to press him against her in a despairing clasp . And they remained , without any further desires , with the past of their unfortunate love , which they had not been able to satisfy . Was it , then , done with for ever ? Would they never dare to love each other some day , now that they were free ? It only needed a little happiness to dissipate their shame--that awkwardness which prevented them from coming together because of all sorts of ideas which they themselves could not read clearly . " Go to bed again , " she whispered . " I do n't want to light up , it would wake mother . It is time ; leave me . " He could not hear ; he was pressing her wildly , with a heart drowned in immense sadness . The need for peace , an irresistible need for happiness , was carrying him away ; and he saw himself married , in a neat little house , with no other ambition than to live and to die there , both of them together . He would be satisfied with bread ; and if there were
  only enough for one , she should have it . What was the good of anything else ? Was there anything in life worth more ? But she was unfolding her naked arms . " Please , leave me . " Then , in a sudden impulse , he said in her ear : " Wait , I 'm coming with you . " And he was himself surprised at what he had said . He had sworn never to go down again ; whence then came this sudden decision , arising from his lips without thought of his , without even a moment 's discussion ? There was now such calm within him , so complete a cure of his doubts , that he persisted like a man saved by chance , who has at last found the only harbour from his torment . So he refused to listen to her when she became alarmed , understanding that he was devoting himself for her and fearing the ill words which would greet him at the pit . He laughed at everything ; the placards promised pardon and that was enough . " I want to work ; that 's my idea . Let us dress and make no noise . " They dressed themse
 lves in the darkness , with a thousand precautions . She had secretly prepared her miner 's clothes the evening before ; he took a jacket and breeches from the cupboard ; and they did not wash themselves for fear of knocking the bowl . All were asleep , but they had to cross the narrow passage where the mother slept . When they started , as ill luck would have it , they stumbled against a chair . She woke and asked drowsily : " Eh ! what is it ? " Catherine had stopped , trembling , and violently pressing �tienne 's hand . " It 's me ; do n't trouble yourself , " he said . " I feel stifled and am going outside to breathe a bit . " " Very well . " And Maheude fell asleep again . Catherine dared not stir . At last she went down into the parlour and divided a slice of bread-and-butter which she had reserved from a loaf given by a Montsou lady . Then they softly closed the door and went away . Souvarine had remained standing near the Avantage , at the corner of the road . For half an h
 our he had been looking at the colliers who were returning to work in the darkness , passing by with the dull tramp of a herd . He was counting them , as a butcher counts his beasts at the entrance to the slaughter-house , and he was surprised at their number ; even his pessimism had not foreseen that the number of cowards would have been so great . The stream continued to pass by , and he grew stiff , very cold , with clenched teeth and bright eyes . But he started . Among the men passing by , whose faces he could not distinguish , he had just recognized one by his walk . He came forward and stopped him . " Where are you going to ? " �tienne , in surprise , instead of replying , stammered : " What ! you 've not set out yet ! " Then he confessed he was going back to the pit . No doubt he had sworn ; only it could not be called life to wait with folded arms for things which would perhaps happen in a hundred years ; and , besides , reasons of his own had decided him . Souvarine had l
 istened to him , shuddering . He seized him by the shoulder , and pushed him towards the settlement . " Go home again ; I want you to . Do you understand ? " But Catherine having approached , he recognized her also . �tienne protested , declaring that he allowed no one to judge his conduct . And the engine-man 's eyes went from the young girl to her companion , while he stepped back with a sudden , relinquishing movement . When there was a woman in a man 's heart , that man was done for ; he might die . Perhaps he saw again in a rapid vision his mistress hanging over there at Moscow that last link cut from his flesh , which had rendered him free of the lives of others and of his own life . He said simply : " Go . " �tienne , feeling awkward , was delaying , and trying to find some friendly word , so as not to separate in this manner . " Then you 're still going ? " " Yes . " " Well , give me your hand , old chap . A pleasant journey , and no ill feeling . " The other stretched out
  an icy hand . Neither friend nor wife . " Good-bye for good this time . " " Yes , good-bye . " And Souvarine , standing motionless in the darkness , watched �tienne and Catherine entering the Voreux . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Marx/74MarxMarxK_ComMan_01_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Marx/74MarxMarxK_ComMan_01_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Marx/74MarxMarxK_ComMan_01_EN.txt.txt
new file mode 100644
index 0000000..c7b7e8c
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Marx/74MarxMarxK_ComMan_01_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Manifesto of the Communist Party A spectre is haunting Europe \u2013 the spectre of communism . All the powers of old Europe have entered into a holy alliance to exorcise this spectre : Pope and Tsar , Metternich and Guizot , French Radicals and German police-spies . Where is the party in opposition that has not been decried as communistic by its opponents in power ? Where is the opposition that has not hurled back the branding reproach of communism , against the more advanced opposition parties , as well as against its reactionary adversaries ? Two things result from this fact : I. Communism is already acknowledged by all European powers to be itself a power . II . It is high time that Communists should openly , in the face of the whole world , publish their views , their aims , their tendencies , and meet this nursery tale of the Spectre of Communism with a manifesto of the party itself . To this end , Communists of various nationalities have assembled in London and sketched the 
 following manifesto , to be published in the English , French , German , Italian , Flemish and Danish languages . I. Bourgeois and Proletarians* The history of all hitherto existing society \u2020 is the history of class struggles . Freeman and slave , patrician and plebeian , lord and serf , guild-master \u2021 and journeyman , in a word , oppressor and oppressed , stood in constant opposition to one another , carried on an uninterrupted , now hidden , now open fight , a fight that each time ended , either in a revolutionary reconstitution of society at large , or in the common ruin of the contending classes . In the earlier epochs of history , we find almost everywhere a complicated arrangement of society into various orders , a manifold gradation of social rank . In ancient Rome we have patricians , knights , plebeians , slaves ; in the Middle Ages , feudal lords , vassals , guild-masters , journeymen , apprentices , serfs ; in almost all of these classes , again , subordinate gradatio
 ns . The modern bourgeois society that has sprouted from the ruins of feudal society has not done away with class antagonisms . It has but established new classes , new conditions of oppression , new forms of struggle in place of the old ones . Our epoch , the epoch of the bourgeoisie , possesses , however , this distinct feature : it has simplified class antagonisms . Society as a whole is more and more splitting up into two great hostile camps , into two great classes directly facing each other \u2013 Bourgeoisie and Proletariat . From the serfs of the Middle Ages sprang the chartered burghers of the earliest towns . From these burgesses the first elements of the bourgeoisie were developed . The discovery of America , the rounding of the Cape , opened up fresh ground for the rising bourgeoisie . The East-Indian and Chinese markets , the colonisation of America , trade with the colonies , the increase in the means of exchange and in commodities generally , gave to commerce , to naviga
 tion , to industry , an impulse never before known , and thereby , to the revolutionary element in the tottering feudal society , a rapid development . The feudal system of industry , in which industrial production was monopolised by closed guilds , now no longer sufficed for the growing wants of the new markets . The manufacturing system took its place . The guild-masters were pushed on one side by the manufacturing middle class ; division of labour between the different corporate guilds vanished in the face of division of labour in each single workshop . Meantime the markets kept ever growing , the demand ever rising . Even manufacturer no longer sufficed . Thereupon , steam and machinery revolutionised industrial production . The place of manufacture was taken by the giant , Modern Industry ; the place of the industrial middle class by industrial millionaires , the leaders of the whole industrial armies , the modern bourgeois . Modern industry has established the world market , f
 or which the discovery of America paved the way . This market has given an immense development to commerce , to navigation , to communication by land . This development has , in its turn , reacted on the extension of industry ; and in proportion as industry , commerce , navigation , railways extended , in the same proportion the bourgeoisie developed , increased its capital , and pushed into the background every class handed down from the Middle Ages . We see , therefore , how the modern bourgeoisie is itself the product of a long course of development , of a series of revolutions in the modes of production and of exchange . Each step in the development of the bourgeoisie was accompanied by a corresponding political advance of that class . An oppressed class under the sway of the feudal nobility , an armed and self-governing association in the medieval commune* : here independent urban republic ( as in Italy and Germany ) ; there taxable \u201c third estate \u201d of the monarchy ( as in 
 France ) ; afterwards , in the period of manufacturing proper , serving either the semi-feudal or the absolute monarchy as a counterpoise against the nobility , and , in fact , cornerstone of the great monarchies in general , the bourgeoisie has at last , since the establishment of Modern Industry and of the world market , conquered for itself , in the modern representative State , exclusive political sway . The executive of the modern state is but a committee for managing the common affairs of the whole bourgeoisie . The bourgeoisie , historically , has played a most revolutionary part . The bourgeoisie , wherever it has got the upper hand , has put an end to all feudal , patriarchal , idyllic relations . It has pitilessly torn asunder the motley feudal ties that bound man to his \u201c natural superiors \u201d , and has left remaining no other nexus between man and man than naked self-interest , than callous \u201c cash payment \u201d . It has drowned the most heavenly ecstasies of religious 
 fervour , of chivalrous enthusiasm , of philistine sentimentalism , in the icy water of egotistical calculation . It has resolved personal worth into exchange value , and in place of the numberless indefeasible chartered freedoms , has set up that single , unconscionable freedom \u2013 Free Trade . In one word , for exploitation , veiled by religious and political illusions , it has substituted naked , shameless , direct , brutal exploitation . The bourgeoisie has stripped of its halo every occupation hitherto honoured and looked up to with reverent awe . It has converted the physician , the lawyer , the priest , the poet , the man of science , into its paid wage labourers . The bourgeoisie has torn away from the family its sentimental veil , and has reduced the family relation to a mere money relation . The bourgeoisie has disclosed how it came to pass that the brutal display of vigour in the Middle Ages , which reactionaries so much admire , found its fitting complement in the most s
 lothful indolence . It has been the first to show what man\u2019s activity can bring about . It has accomplished wonders far surpassing Egyptian pyramids , Roman aqueducts , and Gothic cathedrals ; it has conducted expeditions that put in the shade all former Exoduses of nations and crusades . The bourgeoisie cannot exist without constantly revolutionising the instruments of production , and thereby the relations of production , and with them the whole relations of society . Conservation of the old modes of production in unaltered form , was , on the contrary , the first condition of existence for all earlier industrial classes . Constant revolutionising of production , uninterrupted disturbance of all social conditions , everlasting uncertainty and agitation distinguish the bourgeois epoch from all earlier ones . All fixed , fast-frozen relations , with their train of ancient and venerable prejudices and opinions , are swept away , all new-formed ones become antiquated before they can
  ossify . All that is solid melts into air , all that is holy is profaned , and man is at last compelled to face with sober senses his real conditions of life , and his relations with his kind . The need of a constantly expanding market for its products chases the bourgeoisie over the entire surface of the globe . It must nestle everywhere , settle everywhere , establish connexions everywhere . The bourgeoisie has through its exploitation of the world market given a cosmopolitan character to production and consumption in every country . To the great chagrin of Reactionists , it has drawn from under the feet of industry the national ground on which it stood . All old-established national industries have been destroyed or are daily being destroyed . They are dislodged by new industries , whose introduction becomes a life and death question for all civilised nations , by industries that no longer work up indigenous raw material , but raw material drawn from the remotest zones ; industr
 ies whose products are consumed , not only at home , but in every quarter of the globe . In place of the old wants , satisfied by the production of the country , we find new wants , requiring for their satisfaction the products of distant lands and climes . In place of the old local and national seclusion and self-sufficiency , we have intercourse in every direction , universal inter-dependence of nations . And as in material , so also in intellectual production . The intellectual creations of individual nations become common property . National one-sidedness and narrow-mindedness become more and more impossible , and from the numerous national and local literatures , there arises a world literature . The bourgeoisie , by the rapid improvement of all instruments of production , by the immensely facilitated means of communication , draws all , even the most barbarian , nations into civilisation . The cheap prices of commodities are the heavy artillery with which it batters down all C
 hinese walls , with which it forces the barbarians \u2019 intensely obstinate hatred of foreigners to capitulate . It compels all nations , on pain of extinction , to adopt the bourgeois mode of production ; it compels them to introduce what it calls civilisation into their midst , i. e. , to become bourgeois themselves . In one word , it creates a world after its own image . The bourgeoisie has subjected the country to the rule of the towns . It has created enormous cities , has greatly increased the urban population as compared with the rural , and has thus rescued a considerable part of the population from the idiocy of rural life . Just as it has made the country dependent on the towns , so it has made barbarian and semi-barbarian countries dependent on the civilised ones , nations of peasants on nations of bourgeois , the East on the West . The bourgeoisie keeps more and more doing away with the scattered state of the population , of the means of production , and of property . It 
 has agglomerated population , centralised the means of production , and has concentrated property in a few hands . The necessary consequence of this was political centralisation . Independent , or but loosely connected provinces , with separate interests , laws , governments , and systems of taxation , became lumped together into one nation , with one government , one code of laws , one national class-interest , one frontier , and one customs-tariff . The bourgeoisie , during its rule of scarce one hundred years , has created more massive and more colossal productive forces than have all preceding generations together . Subjection of Nature\u2019s forces to man , machinery , application of chemistry to industry and agriculture , steam-navigation , railways , electric telegraphs , clearing of whole continents for cultivation , canalisation of rivers , whole populations conjured out of the ground \u2013 what earlier century had even a presentiment that such productive forces slumbered in th
 e lap of social labour ? We see then : the means of production and of exchange , on whose foundation the bourgeoisie built itself up , were generated in feudal society . At a certain stage in the development of these means of production and of exchange , the conditions under which feudal society produced and exchanged , the feudal organisation of agriculture and manufacturing industry , in one word , the feudal relations of property became no longer compatible with the already developed productive forces ; they became so many fetters . They had to be burst asunder ; they were burst asunder . Into their place stepped free competition , accompanied by a social and political constitution adapted in it , and the economic and political sway of the bourgeois class . A similar movement is going on before our own eyes . Modern bourgeois society , with its relations of production , of exchange and of property , a society that has conjured up such gigantic means of production and of exchange 
 , is like the sorcerer who is no longer able to control the powers of the nether world whom he has called up by his spells . For many a decade past the history of industry and commerce is but the history of the revolt of modern productive forces against modern conditions of production , against the property relations that are the conditions for the existence of the bourgeois and of its rule . It is enough to mention the commercial crises that by their periodical return put the existence of the entire bourgeois society on its trial , each time more threateningly . In these crises , a great part not only of the existing products , but also of the previously created productive forces , are periodically destroyed . In these crises , there breaks out an epidemic that , in all earlier epochs , would have seemed an absurdity \u2013 the epidemic of over-production . Society suddenly finds itself put back into a state of momentary barbarism ; it appears as if a famine , a universal war of devas
 tation , had cut off the supply of every means of subsistence ; industry and commerce seem to be destroyed ; and why ? Because there is too much civilisation , too much means of subsistence , too much industry , too much commerce . The productive forces at the disposal of society no longer tend to further the development of the conditions of bourgeois property ; on the contrary , they have become too powerful for these conditions , by which they are fettered , and so soon as they overcome these fetters , they bring disorder into the whole of bourgeois society , endanger the existence of bourgeois property . The conditions of bourgeois society are too narrow to comprise the wealth created by them . And how does the bourgeoisie get over these crises ? On the one hand by enforced destruction of a mass of productive forces ; on the other , by the conquest of new markets , and by the more thorough exploitation of the old ones . That is to say , by paving the way for more extensive and mo
 re destructive crises , and by diminishing the means whereby crises are prevented . The weapons with which the bourgeoisie felled feudalism to the ground are now turned against the bourgeoisie itself . But not only has the bourgeoisie forged the weapons that bring death to itself ; it has also called into existence the men who are to wield those weapons \u2013 the modern working class \u2013 the proletarians . In proportion as the bourgeoisie , i. e. , capital , is developed , in the same proportion is the proletariat , the modern working class , developed \u2013 a class of labourers , who live only so long as they find work , and who find work only so long as their labour increases capital . These labourers , who must sell themselves piecemeal , are a commodity , like every other article of commerce , and are consequently exposed to all the vicissitudes of competition , to all the fluctuations of the market . Owing to the extensive use of machinery , and to the division of labour , the work
  of the proletarians has lost all individual character , and , consequently , all charm for the workman . He becomes an appendage of the machine , and it is only the most simple , most monotonous , and most easily acquired knack , that is required of him . Hence , the cost of production of a workman is restricted , almost entirely , to the means of subsistence that he requires for maintenance , and for the propagation of his race . But the price of a commodity , and therefore also of labour , is equal to its cost of production . In proportion , therefore , as the repulsiveness of the work increases , the wage decreases . Nay more , in proportion as the use of machinery and division of labour increases , in the same proportion the burden of toil also increases , whether by prolongation of the working hours , by the increase of the work exacted in a given time or by increased speed of machinery , etc. Modern Industry has converted the little workshop of the patriarchal master into the
  great factory of the industrial capitalist . Masses of labourers , crowded into the factory , are organised like soldiers . As privates of the industrial army they are placed under the command of a perfect hierarchy of officers and sergeants . Not only are they slaves of the bourgeois class , and of the bourgeois State ; they are daily and hourly enslaved by the machine , by the overlooker , and , above all , by the individual bourgeois manufacturer himself . The more openly this despotism proclaims gain to be its end and aim , the more petty , the more hateful and the more embittering it is . The less the skill and exertion of strength implied in manual labour , in other words , the more modern industry becomes developed , the more is the labour of men superseded by that of women . Differences of age and sex have no longer any distinctive social validity for the working class . All are instruments of labour , more or less expensive to use , according to their age and sex . No soon
 er is the exploitation of the labourer by the manufacturer , so far , at an end , that he receives his wages in cash , than he is set upon by the other portions of the bourgeoisie , the landlord , the shopkeeper , the pawnbroker , etc. The lower strata of the middle class \u2013 the small tradespeople , shopkeepers , and retired tradesmen generally , the handicraftsmen and peasants \u2013 all these sink gradually into the proletariat , partly because their diminutive capital does not suffice for the scale on which Modern Industry is carried on , and is swamped in the competition with the large capitalists , partly because their specialised skill is rendered worthless by new methods of production . Thus the proletariat is recruited from all classes of the population . The proletariat goes through various stages of development . With its birth begins its struggle with the bourgeoisie . At first the contest is carried on by individual labourers , then by the workpeople of a factory , then by
  the operative of one trade , in one locality , against the individual bourgeois who directly exploits them . They direct their attacks not against the bourgeois conditions of production , but against the instruments of production themselves ; they destroy imported wares that compete with their labour , they smash to pieces machinery , they set factories ablaze , they seek to restore by force the vanished status of the workman of the Middle Ages . At this stage , the labourers still form an incoherent mass scattered over the whole country , and broken up by their mutual competition . If anywhere they unite to form more compact bodies , this is not yet the consequence of their own active union , but of the union of the bourgeoisie , which class , in order to attain its own political ends , is compelled to set the whole proletariat in motion , and is moreover yet , for a time , able to do so . At this stage , therefore , the proletarians do not fight their enemies , but the enemies of
  their enemies , the remnants of absolute monarchy , the landowners , the non-industrial bourgeois , the petty bourgeois . Thus , the whole historical movement is concentrated in the hands of the bourgeoisie ; every victory so obtained is a victory for the bourgeoisie . But with the development of industry , the proletariat not only increases in number ; it becomes concentrated in greater masses , its strength grows , and it feels that strength more . The various interests and conditions of life within the ranks of the proletariat are more and more equalised , in proportion as machinery obliterates all distinctions of labour , and nearly everywhere reduces wages to the same low level . The growing competition among the bourgeois , and the resulting commercial crises , make the wages of the workers ever more fluctuating . The increasing improvement of machinery , ever more rapidly developing , makes their livelihood more and more precarious ; the collisions between individual workmen
  and individual bourgeois take more and more the character of collisions between two classes . Thereupon , the workers begin to form combinations ( Trades \u2019 Unions ) against the bourgeois ; they club together in order to keep up the rate of wages ; they found permanent associations in order to make provision beforehand for these occasional revolts . Here and there , the contest breaks out into riots . Now and then the workers are victorious , but only for a time . The real fruit of their battles lies , not in the immediate result , but in the ever expanding union of the workers . This union is helped on by the improved means of communication that are created by modern industry , and that place the workers of different localities in contact with one another . It was just this contact that was needed to centralise the numerous local struggles , all of the same character , into one national struggle between classes . But every class struggle is a political struggle . And that union ,
  to attain which the burghers of the Middle Ages , with their miserable highways , required centuries , the modern proletarian , thanks to railways , achieve in a few years . This organisation of the proletarians into a class , and , consequently into a political party , is continually being upset again by the competition between the workers themselves . But it ever rises up again , stronger , firmer , mightier . It compels legislative recognition of particular interests of the workers , by taking advantage of the divisions among the bourgeoisie itself . Thus , the ten-hours \u2019 bill in England was carried . Altogether collisions between the classes of the old society further , in many ways , the course of development of the proletariat . The bourgeoisie finds itself involved in a constant battle . At first with the aristocracy ; later on , with those portions of the bourgeoisie itself , whose interests have become antagonistic to the progress of industry ; at all time with the bour
 geoisie of foreign countries . In all these battles , it sees itself compelled to appeal to the proletariat , to ask for help , and thus , to drag it into the political arena . The bourgeoisie itself , therefore , supplies the proletariat with its own elements of political and general education , in other words , it furnishes the proletariat with weapons for fighting the bourgeoisie . Further , as we have already seen , entire sections of the ruling class are , by the advance of industry , precipitated into the proletariat , or are at least threatened in their conditions of existence . These also supply the proletariat with fresh elements of enlightenment and progress . Finally , in times when the class struggle nears the decisive hour , the progress of dissolution going on within the ruling class , in fact within the whole range of old society , assumes such a violent , glaring character , that a small section of the ruling class cuts itself adrift , and joins the revolutionary cla
 ss , the class that holds the future in its hands . Just as , therefore , at an earlier period , a section of the nobility went over to the bourgeoisie , so now a portion of the bourgeoisie goes over to the proletariat , and in particular , a portion of the bourgeois ideologists , who have raised themselves to the level of comprehending theoretically the historical movement as a whole . Of all the classes that stand face to face with the bourgeoisie today , the proletariat alone is a really revolutionary class . The other classes decay and finally disappear in the face of Modern Industry ; the proletariat is its special and essential product . The lower middle class , the small manufacturer , the shopkeeper , the artisan , the peasant , all these fight against the bourgeoisie , to save from extinction their existence as fractions of the middle class . They are therefore not revolutionary , but conservative . Nay more , they are reactionary , for they try to roll back the wheel of hi
 story . If by chance , they are revolutionary , they are only so in view of their impending transfer into the proletariat ; they thus defend not their present , but their future interests , they desert their own standpoint to place themselves at that of the proletariat . The \u201c dangerous class \u201d , [ lumpenproletariat ] the social scum , that passively rotting mass thrown off by the lowest layers of the old society , may , here and there , be swept into the movement by a proletarian revolution ; its conditions of life , however , prepare it far more for the part of a bribed tool of reactionary intrigue . In the condition of the proletariat , those of old society at large are already virtually swamped . The proletarian is without property ; his relation to his wife and children has no longer anything in common with the bourgeois family relations ; modern industry labour , modern subjection to capital , the same in England as in France , in America as in Germany , has stripped him o
 f every trace of national character . Law , morality , religion , are to him so many bourgeois prejudices , behind which lurk in ambush just as many bourgeois interests . All the preceding classes that got the upper hand sought to fortify their already acquired status by subjecting society at large to their conditions of appropriation . The proletarians cannot become masters of the productive forces of society , except by abolishing their own previous mode of appropriation , and thereby also every other previous mode of appropriation . They have nothing of their own to secure and to fortify ; their mission is to destroy all previous securities for , and insurances of , individual property . All previous historical movements were movements of minorities , or in the interest of minorities . The proletarian movement is the self-conscious , independent movement of the immense majority , in the interest of the immense majority . The proletariat , the lowest stratum of our present society
  , cannot stir , cannot raise itself up , without the whole superincumbent strata of official society being sprung into the air . Though not in substance , yet in form , the struggle of the proletariat with the bourgeoisie is at first a national struggle . The proletariat of each country must , of course , first of all settle matters with its own bourgeoisie . In depicting the most general phases of the development of the proletariat , we traced the more or less veiled civil war , raging within existing society , up to the point where that war breaks out into open revolution , and where the violent overthrow of the bourgeoisie lays the foundation for the sway of the proletariat . Hitherto , every form of society has been based , as we have already seen , on the antagonism of oppressing and oppressed classes . But in order to oppress a class , certain conditions must be assured to it under which it can , at least , continue its slavish existence . The serf , in the period of serfdom 
 , raised himself to membership in the commune , just as the petty bourgeois , under the yoke of the feudal absolutism , managed to develop into a bourgeois . The modern labourer , on the contrary , instead of rising with the process of industry , sinks deeper and deeper below the conditions of existence of his own class . He becomes a pauper , and pauperism develops more rapidly than population and wealth . And here it becomes evident , that the bourgeoisie is unfit any longer to be the ruling class in society , and to impose its conditions of existence upon society as an over-riding law . It is unfit to rule because it is incompetent to assure an existence to its slave within his slavery , because it cannot help letting him sink into such a state , that it has to feed him , instead of being fed by him . Society can no longer live under this bourgeoisie , in other words , its existence is no longer compatible with society . The essential conditions for the existence and for the sway
  of the bourgeois class is the formation and augmentation of capital ; the condition for capital is wage-labour . Wage-labour rests exclusively on competition between the labourers . The advance of industry , whose involuntary promoter is the bourgeoisie , replaces the isolation of the labourers , due to competition , by the revolutionary combination , due to association . The development of Modern Industry , therefore , cuts from under its feet the very foundation on which the bourgeoisie produces and appropriates products . What the bourgeoisie therefore produces , above all , are its own grave-diggers . Its fall and the victory of the proletariat are equally inevitable . 
\ No newline at end of file


[37/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/47FictBalzacH_Goriot_Ib_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/47FictBalzacH_Goriot_Ib_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/47FictBalzacH_Goriot_Ib_EN.txt.txt
new file mode 100644
index 0000000..601d49d
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/47FictBalzacH_Goriot_Ib_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff The next day Rastignac dressed himself very elegantly , and about three o'clock in the afternoon went to call on Mme . de Restaud . On the way thither he indulged in the wild intoxicating dreams which fill a young head so full of delicious excitement . Young men at his age take no account of obstacles nor of dangers ; they see success in every direction ; imagination has free play , and turns their lives into a romance ; they are saddened or discouraged by the collapse of one of the visionary schemes that have no existence save in their heated fancy . If youth were not ignorant and timid , civilization would be impossible . Eugene took unheard-of pains to keep himself in a spotless condition , but on his way through the streets he began to think about Mme . de Restaud and what he should say to her . He equipped himself with wit , rehearsed repartees in the course of an imaginary conversation , and prepared certain neat speeches a la Talleyrand , conjuring up a series of small ev
 ents which should prepare the way for the declaration on which he had based his future ; and during these musings the law student was bespattered with mud , and by the time he reached the Palais Royal he was obliged to have his boots blacked and his trousers brushed . " If I were rich , " he said , as he changed the five-franc piece he had brought with him in case anything might happen , " I would take a cab , then I could think at my ease . " At last he reached the Rue du Helder , and asked for the Comtesse de Restaud . He bore the contemptuous glances of the servants , who had seen him cross the court on foot , with the cold fury of a man who knows that he will succeed some day . He understood the meaning of their glances at once , for he had felt his inferiority as soon as he entered the court , where a smart cab was waiting . All the delights of life in Paris seemed to be implied by this visible and manifest sign of luxury and extravagance . A fine horse , in magnificent harness
  , was pawing the ground , and all at once the law student felt out of humor with himself . Every compartment in his brain which he had thought to find so full of wit was bolted fast ; he grew positively stupid . He sent up his name to the Countess , and waited in the ante-chamber , standing on one foot before a window that looked out upon the court ; mechanically he leaned his elbow against the sash , and stared before him . The time seemed long ; he would have left the house but for the southern tenacity of purpose which works miracles when it is single-minded . " Madame is in her boudoir , and cannot see any one at present , sir , " said the servant . " She gave me no answer ; but if you will go into the dining-room , there is some one already there . " Rastignac was impressed with a sense of the formidable power of the lackey who can accuse or condemn his masters by a word ; he coolly opened the door by which the man had just entered the ante-chamber , meaning , no doubt , to sh
 ow these insolent flunkeys that he was familiar with the house ; but he found that he had thoughtlessly precipitated himself into a small room full of dressers , where lamps were standing , and hot-water pipes , on which towels were being dried ; a dark passage and a back staircase lay beyond it . Stifled laughter from the ante-chamber added to his confusion . " This way to the drawing-room , sir , " said the servant , with the exaggerated respect which seemed to be one more jest at his expense . Eugene turned so quickly that he stumbled against a bath . By good luck , he managed to keep his hat on his head , and saved it from immersion in the water ; but just as he turned , a door opened at the further end of the dark passage , dimly lighted by a small lamp . Rastignac heard voices and the sound of a kiss ; one of the speakers was Mme . de Restaud , the other was Father Goriot . Eugene followed the servant through the dining-room into the drawing-room ; he went to a window that loo
 ked out into the courtyard , and stood there for a while . He meant to know whether this Goriot was really the Goriot that he knew . His heart beat unwontedly fast ; he remembered Vautrin 's hideous insinuations . A well-dressed young man suddenly emerged from the room almost as Eugene entered it , saying impatiently to the servant who stood at the door : " I am going , Maurice . Tell Madame la Comtesse that I waited more than half an hour for her . " Whereupon this insolent being , who , doubtless , had a right to be insolent , sang an Italian trill , and went towards the window where Eugene was standing , moved thereto quite as much by a desire to see the student 's face as by a wish to look out into the courtyard . " But M. le Comte had better wait a moment longer ; madame is disengaged , " said Maurice , as he returned to the ante-chamber . Just at that moment Father Goriot appeared close to the gate ; he had emerged from a door at the foot of the back staircase . The worthy sou
 l was preparing to open his umbrella regardless of the fact that the great gate had opened to admit a tilbury , in which a young man with a ribbon at his button-hole was seated . Father Goriot had scarcely time to start back and save himself . The horse took fright at the umbrella , swerved , and dashed forward towards the flight of steps . The young man looked round in annoyance , saw Father Goriot , and greeted him as he went out with constrained courtesy , such as people usually show to a money-lender so long as they require his services , or the sort of respect they feel it necessary to show for some one whose reputation has been blown upon , so that they blush to acknowledge his acquaintance . Father Goriot gave him a little friendly nod and a good-natured smile . All this happened with lightning speed . Eugene was so deeply interested that he forgot that he was not alone till he suddenly heard the Countess ' voice . " Oh ! Maxime , were you going away ? " she said reproachfull
 y , with a shade of pique in her manner . The Countess had not seen the incident nor the entrance of the tilbury . Rastignac turned abruptly and saw her standing before him , coquettishly dressed in a loose white cashmere gown with knots of rose-colored ribbon here and there ; her hair was carelessly coiled about her head , as is the wont of Parisian women in the morning ; there was a soft fragrance about her--doubtless she was fresh from a bath ; --her graceful form seemed more flexible , her beauty more luxuriant . Her eyes glistened . A young man can see everything at a glance ; he feels the radiant influence of woman as a plant discerns and absorbs its nutriment from the air ; he did not need to touch her hands to feel their cool freshness . He saw faint rose tints through the cashmere of the dressing gown ; it had fallen slightly open , giving glimpses of a bare throat , on which the student 's eyes rested . The Countess had no need of the adventitious aid of corsets ; her gird
 le defined the outlines of her slender waist ; her throat was a challenge to love ; her feet , thrust into slippers , were daintily small . As Maxime took her hand and kissed it , Eugene became aware of Maxime 's existence , and the Countess saw Eugene . " Oh ! is that you M. de Rastignac ? I am very glad to see you , " she said , but there was something in her manner that a shrewd observer would have taken as a hint to depart . Maxime , as the Countess Anastasie had called the young man with the haughty insolence of bearing , looked from Eugene to the lady , and from the lady to Eugene ; it was sufficiently evident that he wished to be rid of the latter . An exact and faithful rendering of the glance might be given in the words : " Look here , my dear ; I hope you intend to send this little whipper-snapper about his business . " The Countess consulted the young man 's face with an intent submissiveness that betrays all the secrets of a woman 's heart , and Rastignac all at once beg
 an to hate him violently . To begin with , the sight of the fair carefully arranged curls on the other 's comely head had convinced him that his own crop was hideous ; Maxime 's boots , moreover , were elegant and spotless , while his own , in spite of all his care , bore some traces of his recent walk ; and , finally , Maxime 's overcoat fitted the outline of his figure gracefully , he looked like a pretty woman , while Eugene was wearing a black coat at half-past two . The quick-witted child of the Charente felt the disadvantage at which he was placed beside this tall , slender dandy , with the clear gaze and the pale face , one of those men who would ruin orphan children without scruple . Mme . de Restaud fled into the next room without waiting for Eugene to speak ; shaking out the skirts of her dressing-gown in her flight , so that she looked like a white butterfly , and Maxime hurried after her . Eugene , in a fury , followed Maxime and the Countess , and the three stood once m
 ore face to face by the hearth in the large drawing-room . The law student felt quite sure that the odious Maxime found him in the way , and even at the risk of displeasing Mme . de Restaud , he meant to annoy the dandy . It had struck him all at once that he had seen the young man before at Mme . de Beauseant 's ball ; he guessed the relation between Maxime and Mme . de Restaud ; and with the youthful audacity that commits prodigious blunders or achieves signal success , he said to himself , " This is my rival ; I mean to cut him out . " Rash resolve ! He did not know that M. le Comte Maxime de Trailles would wait till he was insulted , so as to fire first and kill his man . Eugene was a sportsman and a good shot , but he had not yet hit the bulls 's eye twenty times out of twenty-two . The young Count dropped into a low chair by the hearth , took up the tongs , and made up the fire so violently and so sulkily , that Anastasie 's fair face suddenly clouded over . She turned to Euge
 ne , with a cool , questioning glance that asked plainly , " Why do you not go ? " a glance which well-bred people regard as a cue to make their exit . Eugene assumed an amiable expression . " Madame , " he began , " I hastened to call upon you---- " He stopped short . The door opened , and the owner of the tilbury suddenly appeared . He had left his hat outside , and did not greet the Countess ; he looked meditatively at Rastignac , and held out his hand to Maxime with a cordial " Good morning , " that astonished Eugene not a little . The young provincial did not understand the amenities of a triple alliance . " M. de Restaud , " said the Countess , introducing her husband to the law student . Eugene bowed profoundly . " This gentleman , " she continued , presenting Eugene to her husband , " is M. de Rastignac ; he is related to Mme . la Vicomtesse de Beauseant through the Marcillacs ; I had the pleasure of meeting him at her last ball . " _Related to Mme . la Vicomtesse de Beausea
 nt through the Marcillacs ! _ These words , on which the countess threw ever so slight an emphasis , by reason of the pride that the mistress of a house takes in showing that she only receives people of distinction as visitors in her house , produced a magical effect . The Count 's stiff manner relaxed at once as he returned the student 's bow . " Delighted to have an opportunity of making your acquaintance , " he said . Maxime de Trailles himself gave Eugene an uneasy glance , and suddenly dropped his insolent manner . The mighty name had all the power of a fairy 's wand ; those closed compartments in the southern brain flew open again ; Rastignac 's carefully drilled faculties returned . It was as if a sudden light had pierced the obscurity of this upper world of Paris , and he began to see , though everything was indistinct as yet . Mme . Vauquer 's lodging-house and Father Goriot were very far remote from his thoughts . " I thought that the Marcillacs were extinct , " the Comte 
 de Restaud said , addressing Eugene . " Yes , they are extinct , " answered the law student . " My great-uncle , the Chevalier de Rastignac , married the heiress of the Marcillac family . They had only one daughter , who married the Marechal de Clarimbault , Mme . de Beauseant 's grandfather on the mother 's side . We are the younger branch of the family , and the younger branch is all the poorer because my great-uncle , the Vice-Admiral , lost all that he had in the King 's service . The Government during the Revolution refused to admit our claims when the Compagnie des Indes was liquidated . " " Was not your great-uncle in command of the _Vengeur_ before 1789 ? " " Yes . " " Then he would be acquainted with my grandfather , who commanded the _Warwick_ . " Maxime looked at Mme . de Restaud and shrugged his shoulders , as who should say , " If he is going to discuss nautical matters with that fellow , it is all over with us . " Anastasie understood the glance that M. de Trailles gav
 e her . With a woman 's admirable tact , she began to smile and said : " Come with me , Maxime ; I have something to say to you . We will leave you two gentlemen to sail in company on board the _Warwick_ and the _Vengeur_ . " She rose to her feet and signed to Maxime to follow her , mirth and mischief in her whole attitude , and the two went in the direction of the boudoir . The _morganatic_ couple ( to use a convenient German expression which has no exact equivalent ) had reached the door , when the Count interrupted himself in his talk with Eugene . " Anastasie ! " he cried pettishly , " just stay a moment , dear ; you know very well that---- " " I am coming back in a minute , " she interrupted ; " I have a commission for Maxime to execute , and I want to tell him about it . " She came back almost immediately . She had noticed the inflection in her husband 's voice , and knew that it would not be safe to retire to the boudoir ; like all women who are compelled to study their husba
 nds ' characters in order to have their own way , and whose business it is to know exactly how far they can go without endangering a good understanding , she was very careful to avoid petty collisions in domestic life . It was Eugene who had brought about this untoward incident ; so the Countess looked at Maxime and indicated the law student with an air of exasperation . M. de Trailles addressed the Count , the Countess , and Eugene with the pointed remark , " You are busy , I do not want to interrupt you ; good-day , " and he went . " Just wait a moment , Maxime ! " the Count called after him . " Come and dine with us , " said the Countess , leaving Eugene and her husband together once more . She followed Maxime into the little drawing-room , where they sat together sufficiently long to feel sure that Rastignac had taken his leave . The law student heard their laughter , and their voices , and the pauses in their talk ; he grew malicious , exerted his conversational powers for M. d
 e Restaud , flattered him , and drew him into discussions , to the end that he might see the Countess again and discover the nature of her relations with Father Goriot . This Countess with a husband and a lover , for Maxime clearly was her lover , was a mystery . What was the secret tie that bound her to the old tradesman ? This mystery he meant to penetrate , hoping by its means to gain a sovereign ascendency over this fair typical Parisian . " Anastasie ! " the Count called again to his wife . " Poor Maxime ! " she said , addressing the young man . " Come , we must resign ourselves . This evening---- " " I hope , Nasie , " he said in her ear , " that you will give orders not to admit that youngster , whose eyes light up like live coals when he looks at you . He will make you a declaration , and compromise you , and then you will compel me to kill him . " " Are you mad , Maxime ? " she said . " A young lad of a student is , on the contrary , a capital lightning-conductor ; is not t
 hat so ? Of course , I mean to make Restaud furiously jealous of him . " Maxime burst out laughing , and went out , followed by the Countess , who stood at the window to watch him into his carriage ; he shook his whip , and made his horse prance . She only returned when the great gate had been closed after him . " What do you think , dear ? " cried the Count , her husband , " this gentleman 's family estate is not far from Verteuil , on the Charente ; his great-uncle and my grandfather were acquainted . " " Delighted to find that we have acquaintances in common , " said the Countess , with a preoccupied manner . " More than you think , " said Eugene , in a low voice . " What do you mean ? " she asked quickly . " Why , only just now , " said the student , " I saw a gentleman go out at the gate , Father Goriot , my next door neighbor in the house where I am lodging . " At the sound of this name , and the prefix that embellished it , the Count , who was stirring the fire , let the tong
 s fall as though they had burned his fingers , and rose to his feet . " Sir , " he cried , " you might have called him 'Monsieur Goriot ' ! " The Countess turned pale at first at the sight of her husband 's vexation , then she reddened ; clearly she was embarrassed , her answer was made in a tone that she tried to make natural , and with an air of assumed carelessness : " You could not know any one who is dearer to us both ... " She broke off , glanced at the piano as if some fancy had crossed her mind , and asked , " Are you fond of music , M. de Rastignac ? " " Exceedingly , " answered Eugene , flushing , and disconcerted by a dim suspicion that he had somehow been guilty of a clumsy piece of folly . " Do you sing ? " she cried , going to the piano , and , sitting down before it , she swept her fingers over the keyboard from end to end . R-r-r-rah ! " No , madame . " The Comte de Restaud walked to and fro . " That is a pity ; you are without one great means of success . --_Ca-ro ,
  ca-a-ro , ca-a-a-ro , non du-bi-ta-re_ , " sang the Countess . Eugene had a second time waved a magic wand when he uttered Goriot 's name , but the effect seemed to be entirely opposite to that produced by the formula " related to Mme . de Beauseant . " His position was not unlike that of some visitor permitted as a favor to inspect a private collection of curiosities , when by inadvertence he comes into collision with a glass case full of sculptured figures , and three or four heads , imperfectly secured , fall at the shock . He wished the earth would open and swallow him . Mme . de Restaud 's expression was reserved and chilly , her eyes had grown indifferent , and sedulously avoided meeting those of the unlucky student of law . " Madame , " he said , " you wish to talk with M. de Restaud ; permit me to wish you good-day---- " The Countess interrupted him by a gesture , saying hastily , " Whenever you come to see us , both M. de Restaud and I shall be delighted to see you . " Eug
 ene made a profound bow and took his leave , followed by M. de Restaud , who insisted , in spite of his remonstrances , on accompanying him into the hall . " Neither your mistress nor I are at home to that gentleman when he calls , " the Count said to Maurice . As Eugene set foot on the steps , he saw that it was raining . " Come , " said he to himself , " somehow I have just made a mess of it , I do not know how . And now I am going to spoil my hat and coat into the bargain . I ought to stop in my corner , grind away at law , and never look to be anything but a boorish country magistrate . How can I go into society , when to manage properly you want a lot of cabs , varnished boots , gold watch chains , and all sorts of things ; you have to wear white doeskin gloves that cost six francs in the morning , and primrose kid gloves every evening ? A fig for that old humbug of a Goriot ! " When he reached the street door , the driver of a hackney coach , who had probably just deposited a 
 wedding party at their door , and asked nothing better than a chance of making a little money for himself without his employer 's knowledge , saw that Eugene had no umbrella , remarked his black coat , white waistcoat , yellow gloves , and varnished boots , and stopped and looked at him inquiringly . Eugene , in the blind desperation that drives a young man to plunge deeper and deeper into an abyss , as if he might hope to find a fortunate issue in its lowest depths , nodded in reply to the driver 's signal , and stepped into the cab ; a few stray petals of orange blossom and scraps of wire bore witness to its recent occupation by a wedding party . " Where am I to drive , sir ? " demanded the man , who , by this time , had taken off his white gloves . " Confound it ! " Eugene said to himself , " I am in for it now , and at least I will not spend cab-hire for nothing ! --Drive to the Hotel Beauseant , " he said aloud . " Which ? " asked the man , a portentous word that reduced Eugene
  to confusion . This young man of fashion , species incerta , did not know that there were two Hotels Beauseant ; he was not aware how rich he was in relations who did not care about him . " The Vicomte de Beauseant , Rue---- " " De Grenelle , " interrupted the driver , with a jerk of his head . " You see , there are the hotels of the Marquis and Comte de Beauseant in the Rue Saint-Dominique , " he added , drawing up the step . " I know all about that , " said Eugene , severely . --"Everybody is laughing at me to-day , it seems ! " he said to himself , as he deposited his hat on the opposite seat . " This escapade will cost me a king 's ransom , but , at any rate , I shall call on my so-called cousin in a thoroughly aristocratic fashion . Goriot has cost me ten francs already , the old scoundrel . My word ! I will tell Mme . de Beauseant about my adventure ; perhaps it may amuse her . Doubtless she will know the secret of the criminal relation between that handsome woman and the old
  rat without a tail . It would be better to find favor in my cousin 's eyes than to come in contact with that shameless woman , who seems to me to have very expensive tastes . Surely the beautiful Vicomtesse 's personal interest would turn the scale for me , when the mere mention of her name produces such an effect . Let us look higher . If you set yourself to carry the heights of heaven , you must face God . " The innumerable thoughts that surged through his brain might be summed up in these phrases . He grew calmer , and recovered something of his assurance as he watched the falling rain . He told himself that though he was about to squander two of the precious five-franc pieces that remained to him , the money was well laid out in preserving his coat , boots , and hat ; and his cabman 's cry of " Gate , if you please , " almost put him in spirits . A Swiss , in scarlet and gold , appeared , the great door groaned on its hinges , and Rastignac , with sweet satisfaction , beheld hi
 s equipage pass under the archway and stop before the flight of steps beneath the awning . The driver , in a blue-and-red greatcoat , dismounted and let down the step . As Eugene stepped out of the cab , he heard smothered laughter from the peristyle . Three or four lackeys were making merry over the festal appearance of the vehicle . In another moment the law student was enlightened as to the cause of their hilarity ; he felt the full force of the contrast between his equipage and one of the smartest broughams in Paris ; a coachman , with powdered hair , seemed to find it difficult to hold a pair of spirited horses , who stood chafing the bit . In Mme . de Restaud 's courtyard , in the Chaussee d'Antin , he had seen the neat turnout of a young man of six-and-twenty ; in the Faubourg Saint-Germain he found the luxurious equipage of a man of rank ; thirty thousand francs would not have purchased it . " Who can be here ? " said Eugene to himself . He began to understand , though somew
 hat tardily , that he must not expect to find many women in Paris who were not already appropriated , and that the capture of one of these queens would be likely to cost something more than bloodshed . " Confound it all ! I expect my cousin also has her Maxime . " He went up the steps , feeling that he was a blighted being . The glass door was opened for him ; the servants were as solemn as jackasses under the curry comb . So far , Eugene had only been in the ballroom on the ground floor of the Hotel Beauseant ; the fete had followed so closely on the invitation , that he had not had time to call on his cousin , and had therefore never seen Mme . de Beauseant 's apartments ; he was about to behold for the first time a great lady among the wonderful and elegant surroundings that reveal her character and reflect her daily life . He was the more curious , because Mme . de Restaud 's drawing-room had provided him with a standard of comparison . At half-past four the Vicomtesse de Beause
 ant was visible . Five minutes earlier she would not have received her cousin , but Eugene knew nothing of the recognized routine of various houses in Paris . He was conducted up the wide , white-painted , crimson-carpeted staircase , between the gilded balusters and masses of flowering plants , to Mme . de Beauseant 's apartments . He did not know the rumor current about Mme . de Beauseant , one of the biographies told , with variations , in whispers , every evening in the salons of Paris . For three years past her name had been spoken of in connection with that of one of the most wealthy and distinguished Portuguese nobles , the Marquis d'Ajuda-Pinto . It was one of those innocent _liaisons_ which possess so much charm for the two thus attached to each other that they find the presence of a third person intolerable . The Vicomte de Beauseant , therefore , had himself set an example to the rest of the world by respecting , with as good a grace as might be , this morganatic union . 
 Any one who came to call on the Vicomtesse in the early days of this friendship was sure to find the Marquis d'Ajuda-Pinto there . As , under the circumstances , Mme . de Beauseant could not very well shut her door against these visitors , she gave them such a cold reception , and showed so much interest in the study of the ceiling , that no one could fail to understand how much he bored her ; and when it became known in Paris that Mme . de Beauseant was bored by callers between two and four o'clock , she was left in perfect solitude during that interval . She went to the Bouffons or to the Opera with M. de Beauseant and M. d'Ajuda-Pinto ; and M. de Beauseant , like a well-bred man of the world , always left his wife and the Portuguese as soon as he had installed them . But M. d'Ajuda-Pinto must marry , and a Mlle . de Rochefide was the young lady . In the whole fashionable world there was but one person who as yet knew nothing of the arrangement , and that was Mme . de Beauseant . 
 Some of her friends had hinted at the possibility , and she had laughed at them , believing that envy had prompted those ladies to try to make mischief . And now , though the bans were about to be published , and although the handsome Portuguese had come that day to break the news to the Vicomtesse , he had not found courage as yet to say one word about his treachery . How was it ? Nothing is doubtless more difficult than the notification of an ultimatum of this kind . There are men who feel more at their ease when they stand up before another man who threatens their lives with sword or pistol than in the presence of a woman who , after two hours of lamentations and reproaches , falls into a dead swoon and requires salts . At this moment , therefore , M. d'Ajuda-Pinto was on thorns , and anxious to take his leave . He told himself that in some way or other the news would reach Mme . de Beauseant ; he would write , it would be much better to do it by letter , and not to utter the wor
 ds that should stab her to the heart . So when the servant announced M. Eugene de Rastignac , the Marquis d'Ajuda-Pinto trembled with joy . To be sure , a loving woman shows even more ingenuity in inventing doubts of her lover than in varying the monotony of his happiness ; and when she is about to be forsaken , she instinctively interprets every gesture as rapidly as Virgil 's courser detected the presence of his companion by snuffing the breeze . It was impossible , therefore , that Mme . de Beauseant should not detect that involuntary thrill of satisfaction ; slight though it was , it was appalling in its artlessness . Eugene had yet to learn that no one in Paris should present himself in any house without first making himself acquainted with the whole history of its owner , and of its owner 's wife and family , so that he may avoid making any of the terrible blunders which in Poland draw forth the picturesque exclamation , " Harness five bullocks to your cart ! " probably becaus
 e you will need them all to pull you out of the quagmire into which a false step has plunged you . If , down to the present day , our language has no name for these conversational disasters , it is probably because they are believed to be impossible , the publicity given in Paris to every scandal is so prodigious . After the awkward incident at Mme . de Restaud 's , no one but Eugene could have reappeared in his character of bullock-driver in Mme . de Beauseant 's drawing-room . But if Mme . de Restaud and M. de Trailles had found him horribly in the way , M. d'Ajuda hailed his coming with relief . " Good-bye , " said the Portuguese , hurrying to the door , as Eugene made his entrance into a dainty little pink-and-gray drawing-room , where luxury seemed nothing more than good taste . " Until this evening , " said Mme . de Beauseant , turning her head to give the Marquis a glance . " We are going to the Bouffons , are we not ? " " I cannot go , " he said , with his fingers on the doo
 r handle . Mme . de Beauseant rose and beckoned to him to return . She did not pay the slightest attention to Eugene , who stood there dazzled by the sparkling marvels around him ; he began to think that this was some story out of the Arabian Nights made real , and did not know where to hide himself , when the woman before him seemed to be unconscious of his existence . The Vicomtesse had raised the forefinger of her right hand , and gracefully signed to the Marquis to seat himself beside her . The Marquis felt the imperious sway of passion in her gesture ; he came back towards her . Eugene watched him , not without a feeling of envy . " That is the owner of the brougham ! " he said to himself . " But is it necessary to have a pair of spirited horses , servants in livery , and torrents of gold to draw a glance from a woman here in Paris ? " The demon of luxury gnawed at his heart , greed burned in his veins , his throat was parched with the thirst of gold . He had a hundred and thir
 ty francs every quarter . His father , mother , brothers , sisters , and aunt did not spend two hundred francs a month among them . This swift comparison between his present condition and the aims he had in view helped to benumb his faculties . " Why not ? " the Vicomtesse was saying , as she smiled at the Portuguese . " Why cannot you come to the Italiens ? " " Affairs ! I am to dine with the English Ambassador . " " Throw him over . " When a man once enters on a course of deception , he is compelled to add lie to lie . M. d'Ajuda therefore said , smiling , " Do you lay your commands on me ? " " Yes , certainly . " " That was what I wanted to have you say to me , " he answered , dissembling his feelings in a glance which would have reassured any other woman . He took the Vicomtesse 's hand , kissed it , and went . Eugene ran his fingers through his hair , and constrained himself to bow . He thought that now Mme . de Beauseant would give him her attention ; but suddenly she sprang f
 orward , rushed to a window in the gallery , and watched M. d'Ajuda step into his carriage ; she listened to the order that he gave , and heard the Swiss repeat it to the coachman : " To M. de Rochefide 's house . " Those words , and the way in which M. d'Ajuda flung himself back in the carriage , were like a lightning flash and a thunderbolt for her ; she walked back again with a deadly fear gnawing at her heart . The most terrible catastrophes only happen among the heights . The Vicomtesse went to her own room , sat down at a table , and took up a sheet of dainty notepaper . " When , instead of dining with the English Ambassador , " she wrote , " you go to the Rochefides , you owe me an explanation , which I am waiting to hear . " She retraced several of the letters , for her hand was trembling so that they were indistinct ; then she signed the note with an initial C for " Claire de Bourgogne , " and rang the bell . " Jacques , " she said to the servant , who appeared immediately 
 , " take this note to M. de Rochefide 's house at half-past seven and ask for the Marquis d'Ajuda . If M. d'Ajuda is there , leave the note without waiting for an answer ; if he is not there , bring the note back to me . " " Madame la Vicomtess , there is a visitor in the drawing-room . " " Ah ! yes , of course , " she said , opening the door . Eugene was beginning to feel very uncomfortable , but at last the Vicomtesse appeared ; she spoke to him , and the tremulous tones of her voice vibrated through his heart . " Pardon me , monsieur , " she said ; " I had a letter to write . Now I am quite at liberty . " She scarcely knew what she was saying , for even as she spoke she thought , " Ah ! he means to marry Mlle . de Rochefide ? But is he still free ? This evening the marriage shall be broken off , or else ... But before to-morrow I shall know . " " Cousin ... " the student replied . " Eh ? " said the Countess , with an insolent glance that sent a cold shudder through Eugene ; he un
 derstood what that " Eh ? " meant ; he had learned a great deal in three hours , and his wits were on the alert . He reddened : " Madame ... " he began ; he hesitated a moment , and then went on . " Pardon me ; I am in such need of protection that the nearest scrap of relationship could do me no harm . " Mme . de Beauseant smiled but there was sadness in her smile ; even now she felt forebodings of the coming pain , the air she breathed was heavy with the storm that was about to burst . " If you knew how my family are situated , " he went on , " you would love to play the part of a beneficent fairy godmother who graciously clears the obstacles from the path of her protege . " " Well , cousin , " she said , laughing , " and how can I be of service to you ? " " But do I know even that ? I am distantly related to you , and this obscure and remote relationship is even now a perfect godsend to me . You have confused my ideas ; I cannot remember the things that I meant to say to you . I k
 now no one else here in Paris ... . Ah ! if I could only ask you to counsel me , ask you to look upon me as a poor child who would fain cling to the hem of your dress , who would lay down his life for you . " " Would you kill a man for me ? " " Two , " said Eugene . " You , child . Yes , you are a child , " she said , keeping back the tears that came to her eyes ; " you would love sincerely . " " Oh ! " he cried , flinging up his head . The audacity of the student 's answer interested the Vicomtesse in him . The southern brain was beginning to scheme for the first time . Between Mme . de Restaud 's blue boudoir and Mme . de Beauseant 's rose-colored drawing-room he had made a three years ' advance in a kind of law which is not a recognized study in Paris , although it is a sort of higher jurisprudence , and , when well understood , is a highroad to success of every kind . " Ah ! that is what I meant to say ! " said Eugene . " I met Mme . de Restaud at your ball , and this morning I 
 went to see her . " You must have been very much in the way , " said Mme . de Beauseant , smiling as she spoke . " Yes , indeed . I am a novice , and my blunders will set every one against me , if you do not give me your counsel . I believe that in Paris it is very difficult to meet with a young , beautiful , and wealthy woman of fashion who would be willing to teach me , what you women can explain so well--life . I shall find a M. de Trailles everywhere . So I have come to you to ask you to give me a key to a puzzle , to entreat you to tell me what sort of blunder I made this morning . I mentioned an old man---- " " Madame la Duchess de Langeais , " Jacques cut the student short ; Eugene gave expression to his intense annoyance by a gesture . " If you mean to succeed , " said the Vicomtesse in a low voice , " in the first place you must not be so demonstrative . " " Ah ! good morning , dear , " she continued , and rising and crossing the room , she grasped the Duchess ' hands as af
 fectionately as if they had been sisters ; the Duchess responded in the prettiest and most gracious way . " Two intimate friends ! " said Rastignac to himself . " Henceforward I shall have two protectresses ; those two women are great friends , no doubt , and this newcomer will doubtless interest herself in her friend 's cousin . " " To what happy inspiration do I owe this piece of good fortune , dear Antoinette ? " asked Mme . de Beauseant . " Well , I saw M. d'Ajuda-Pinto at M. de Rochefide 's door , so I thought that if I came I should find you alone . " Mme . de Beauseant 's mouth did not tighten , her color did not rise , her expression did not alter , or rather , her brow seemed to clear as the Duchess uttered those deadly words . " If I had known that you were engaged---- " the speaker added , glancing at Eugene . " This gentleman is M. Eugene de Rastignac , one of my cousins , " said the Vicomtesse . " Have you any news of General de Montriveau ? " she continued . " Serizy t
 old me yesterday that he never goes anywhere now ; has he been to see you to-day ? " It was believed that the Duchess was desperately in love with M. de Montriveau , and that he was a faithless lover ; she felt the question in her very heart , and her face flushed as she answered : " He was at the Elysee yesterday . " " In attendance ? " " Claire , " returned the Duchess , and hatred overflowed in the glances she threw at Mme . de Beauseant ; " of course you know that M. d'Ajuda-Pinto is going to marry Mlle . de Rochefide ; the bans will be published to-morrow . " This thrust was too cruel ; the Vicomtesse 's face grew white , but she answered , laughing , " One of those rumors that fools amuse themselves with . What should induce M. d'Ajuda to take one of the noblest names in Portugal to the Rochefides ? The Rochefides were only ennobled yesterday . " " But Bertha will have two hundred thousand livres a year , they say . " " M. d'Ajuda is too wealthy to marry for money . " " But , 
 my dear , Mlle . de Rochefide is a charming girl . " " Indeed ? " " And , as a matter of fact , he is dining with them to-day ; the thing is settled . It is very surprising to me that you should know so little about it . " Mme . de Beauseant turned to Rastignac . " What was the blunder that you made , monsieur ? " she asked . " The poor boy is only just launched into the world , Antoinette , so that he understands nothing of all this that we are speaking of . Be merciful to him , and let us finish our talk to-morrow . Everything will be announced to-morrow , you know , and your kind informal communication can be accompanied by official confirmation . " The Duchess gave Eugene one of those insolent glances that measure a man from head to foot , and leave him crushed and annihilated . " Madame , I have unwittingly plunged a dagger into Mme . de Restaud 's heart ; unwittingly--therein lies my offence , " said the student of law , whose keen brain had served him sufficiently well , for 
 he had detected the biting epigrams that lurked beneath this friendly talk . " You continue to receive , possibly you fear , those who know the amount of pain that they deliberately inflict ; but a clumsy blunderer who has no idea how deeply he wounds is looked upon as a fool who does not know how to make use of his opportunities , and every one despises him . " Mme . de Beauseant gave the student a glance , one of those glances in which a great soul can mingle dignity and gratitude . It was like balm to the law student , who was still smarting under the Duchess ' insolent scrutiny ; she had looked at him as an auctioneer might look at some article to appraise its value . " Imagine , too , that I had just made some progress with the Comte de Restaud ; for I should tell you , madame , " he went on , turning to the Duchess with a mixture of humility and malice in his manner , " that as yet I am only a poor devil of a student , very much alone in the world , and very poor---- " " You s
 hould not tell us that , M. de Rastignac . We women never care about anything that no one else will take . " " Bah ! " said Eugene . " I am only two-and-twenty , and I must make up my mind to the drawbacks of my time of life . Besides , I am confessing my sins , and it would be impossible to kneel in a more charming confessional ; you commit your sins in one drawing-room , and receive absolution for them in another . " The Duchess ' expression grew colder , she did not like the flippant tone of these remarks , and showed that she considered them to be in bad taste by turning to the Vicomtesse with--"This gentleman has only just come---- " Mme . de Beauseant began to laugh outright at her cousin and at the Duchess both . " He has only just come to Paris , dear , and is in search of some one who will give him lessons in good taste . " " Mme . la Duchesse , " said Eugene , " is it not natural to wish to be initiated into the mysteries which charm us ? " ( " Come , now , " he said to hi
 mself , " my language is superfinely elegant , I 'm sure . " ) " But Mme . de Restaud is herself , I believe , M. de Trailles ' pupil , " said the Duchess . " Of that I had no idea , madame , " answered the law student , " so I rashly came between them . In fact , I got on very well with the lady 's husband , and his wife tolerated me for a time until I took it into my head to tell them that I knew some one of whom I had just caught a glimpse as he went out by a back staircase , a man who had given the Countess a kiss at the end of a passage . " " Who was it ? " both women asked together . " An old man who lives at the rate of two louis a month in the Faubourg Saint-Marceau , where I , a poor student , lodge likewise . He is a truly unfortunate creature , everybody laughs at him--we all call him 'Father Goriot . ' " " Why , child that you are , " cried the Vicomtesse , " Mme . de Restaud was a Mlle . Goriot ! " " The daughter of a vermicelli manufacturer , " the Duchess added ; " an
 d when the little creature went to Court , the daughter of a pastry-cook was presented on the same day . Do you remember , Claire ? The King began to laugh , and made some joke in Latin about flour . People--what was it ? --people---- " " _Ejusdem farinoe_ , " said Eugene . " Yes , that was it , " said the Duchess . " Oh ! is that her father ? " the law student continued , aghast . " Yes , certainly ; the old man had two daughters ; he dotes on them , so to speak , though they will scarcely acknowledge him . " " Did n't the second daughter marry a banker with a German name ? " the Vicomtesse asked , turning to Mme . de Langeais , " a Baron de Nucingen ? And her name is Delphine , is it not ? Is n't she a fair-haired woman who has a side-box at the Opera ? She comes sometimes to the Bouffons , and laughs loudly to attract attention . " The Duchess smiled and said : " I wonder at you , dear . Why do you take so much interest in people of that kind ? One must have been as madly in love
  as Restaud was , to be infatuated with Mlle . Anastasie and her flour sacks . Oh ! he will not find her a good bargain ! She is in M. de Trailles ' hands , and he will ruin her . " " And they do not acknowledge their father ! " Eugene repeated . " Oh ! well , yes , their father , the father , a father , " replied the Vicomtesse , " a kind father who gave them each five or six hundred thousand francs , it is said , to secure their happiness by marrying them well ; while he only kept eight or ten thousand livres a year for himself , thinking that his daughters would always be his daughters , thinking that in them he would live his life twice over again , that in their houses he should find two homes , where he would be loved and looked up to , and made much of . And in two years ' time both his sons-in-law had turned him out of their houses as if he were one of the lowest outcasts . " Tears came into Eugene 's eyes . He was still under the spell of youthful beliefs , he had just left
  home , pure and sacred feelings had been stirred within him , and this was his first day on the battlefield of civilization in Paris . Genuine feeling is so infectious that for a moment the three looked at each other in silence . " _Eh , mon Dieu ! _ " said Mme . de Langeais ; " yes , it seems very horrible , and yet we see such things every day . Is there not a reason for it ? Tell me , dear , have you ever really thought what a son-in-law is ? A son-in-law is the man for whom we bring up , you and I , a dear little one , bound to us very closely in innumerable ways ; for seventeen years she will be the joy of her family , its 'white soul , ' as Lamartine says , and suddenly she will become its scourge . When HE comes and takes her from us , his love from the very beginning is like an axe laid to the root of all the old affection in our darling 's heart , and all the ties that bound her to her family are severed . But yesterday our little daughter thought of no one but her mother 
 and father , as we had no thought that was not for her ; by to-morrow she will have become a hostile stranger . The tragedy is always going on under our eyes . On the one hand you see a father who has sacrificed himself to his son , and his daughter-in-law shows him the last degree of insolence . On the other hand , it is the son-in-law who turns his wife 's mother out of the house . I sometimes hear it said that there is nothing dramatic about society in these days ; but the Drama of the Son-in-law is appalling , to say nothing of our marriages , which have come to be very poor farces . I can explain how it all came about in the old vermicelli maker 's case . I think I recollect that Foriot---- " " Goriot , madame . " " Yes , that Moriot was once President of his Section during the Revolution . He was in the secret of the famous scarcity of grain , and laid the foundation of his fortune in those days by selling flour for ten times its cost . He had as much flour as he wanted . My g
 randmother 's steward sold him immense quantities . No doubt Noriot shared the plunder with the Committee of Public Salvation , as that sort of person always did . I recollect the steward telling my grandmother that she might live at Grandvilliers in complete security , because her corn was as good as a certificate of civism . Well , then , this Loriot , who sold corn to those butchers , has never had but one passion , they say--he idolizes his daughters . He settled one of them under Restaud 's roof , and grafted the other into the Nucingen family tree , the Baron de Nucingen being a rich banker who had turned Royalist . You can quite understand that so long as Bonaparte was Emperor , the two sons-in-law could manage to put up with the old Ninety-three ; but after the restoration of the Bourbons , M. de Restaud felt bored by the old man 's society , and the banker was still more tired of it . His daughters were still fond of him ; they wanted 'to keep the goat and the cabbage , ' s
 o they used to see Joriot whenever there was no one there , under pretence of affection . 'Come to-day , papa , we shall have you all to ourselves , and that will be much nicer ! ' and all that sort of thing . As for me , dear , I believe that love has second-sight : poor Ninety-three ; his heart must have bled . He saw that his daughters were ashamed of him , that if they loved their husbands his visits must make mischief . So he immolated himself . He made the sacrifice because he was a father ; he went into voluntary exile . His daughters were satisfied , so he thought that he had done the best thing he could ; but it was a family crime , and father and daughters were accomplices . You see this sort of thing everywhere . What could this old Doriot have been but a splash of mud in his daughters ' drawing-rooms ? He would only have been in the way , and bored other people , besides being bored himself . And this that happened between father and daughters may happen to the prettiest
  woman in Paris and the man she loves the best ; if her love grows tiresome , he will go ; he will descend to the basest trickery to leave her . It is the same with all love and friendship . Our heart is a treasury ; if you pour out all its wealth at once , you are bankrupt . We show no more mercy to the affection that reveals its utmost extent than we do to another kind of prodigal who has not a penny left . Their father had given them all he had . For twenty years he had given his whole heart to them ; then , one day , he gave them all his fortune too . The lemon was squeezed ; the girls left the rest in the gutter . " " The world is very base , " said the Vicomtesse , plucking at the threads of her shawl . She did not raise her head as she spoke ; the words that Mme . de Langeais had meant for her in the course of her story had cut her to the quick . " Base ? Oh , no , " answered the Duchess ; " the world goes its own way , that is all . If I speak in this way , it is only to sho
 w that I am not duped by it . I think as you do , " she said , pressing the Vicomtesse 's hand . " The world is a slough ; let us try to live on the heights above it . " She rose to her feet and kissed Mme . de Beauseant on the forehead as she said : " You look very charming to-day , dear . I have never seen such a lovely color in your cheeks before . " Then she went out with a slight inclination of the head to the cousin . " Father Goriot is sublime ! " said Eugene to himself , as he remembered how he had watched his neighbor work the silver vessel into a shapeless mass that night . Mme . de Beauseant did not hear him ; she was absorbed in her own thoughts . For several minutes the silence remained unbroken till the law student became almost paralyzed with embarrassment , and was equally afraid to go or stay or speak a word . " The world is basely ungrateful and ill-natured , " said the Vicomtesse at last . " No sooner does a trouble befall you than a friend is ready to bring the t
 idings and to probe your heart with the point of a dagger while calling on you to admire the handle . Epigrams and sarcasms already ! Ah ! I will defend myself ! " She raised her head like the great lady that she was , and lightnings flashed from her proud eyes . " Ah ! " she said , as she saw Eugene , " are you there ? " " Still , " he said piteously . " Well , then , M. de Rastignac , deal with the world as it deserves . You are determined to succeed ? I will help you . You shall sound the depths of corruption in woman ; you shall measure the extent of man 's pitiful vanity . Deeply as I am versed in such learning , there were pages in the book of life that I had not read . Now I know all . The more cold-blooded your calculations , the further you will go . Strike ruthlessly ; you will be feared . Men and women for you must be nothing more than post-horses ; take a fresh relay , and leave the last to drop by the roadside ; in this way you will reach the goal of your ambition . You
  will be nothing here , you see , unless a woman interests herself in you ; and she must be young and wealthy , and a woman of the world . Yet , if you have a heart , lock it carefully away like a treasure ; do not let any one suspect it , or you will be lost ; you would cease to be the executioner , you would take the victim 's place . And if ever you should love , never let your secret escape you ! Trust no one until you are very sure of the heart to which you open your heart . Learn to mistrust every one ; take every precaution for the sake of the love which does not exist as yet . Listen , Miguel"--the name slipped from her so naturally that she did not notice her mistake--"there is something still more appalling than the ingratitude of daughters who have cast off their old father and wish that he were dead , and that is a rivalry between two sisters . Restaud comes of a good family , his wife has been received into their circle ; she has been presented at court ; and her sister
  , her wealthy sister , Mme . Delphine de Nucingen , the wife of a great capitalist , is consumed with envy , and ready to die of spleen . There is gulf set between the sisters--indeed , they are sisters no longer--the two women who refuse to acknowledge their father do not acknowledge each other . So Mme . de Nucingen would lap up all the mud that lies between the Rue Saint-Lazare and the Rue de Grenelle to gain admittance to my salon . She fancied that she should gain her end through de Marsay ; she has made herself de Marsay 's slave , and she bores him . De Marsay cares very little about her . If you will introduce her to me , you will be her darling , her Benjamin ; she will idolize you . If , after that , you can love her , do so ; if not , make her useful . I will ask her to come once or twice to one of my great crushes , but I will never receive her here in the morning . I will bow to her when I see her , and that will be quite sufficient . You have shut the Comtesse de Rest
 aud 's door against you by mentioning Father Goriot 's name . Yes , my good friend , you may call at her house twenty times , and every time out of the twenty you will find that she is not at home . The servants have their orders , and will not admit you . Very well , then , now let Father Goriot gain the right of entry into her sister 's house for you . The beautiful Mme . de Nucingen will give the signal for a battle . As soon as she singles you out , other women will begin to lose their heads about you , and her enemies and rivals and intimate friends will all try to take you from her . There are women who will fall in love with a man because another woman has chosen him ; like the city madams , poor things , who copy our millinery , and hope thereby to acquire our manners . You will have a success , and in Paris success is everything ; it is the key of power . If the women credit you with wit and talent , the men will follow suit so long as you do not undeceive them yourself . T
 here will be nothing you may not aspire to ; you will go everywhere , and you will find out what the world is--an assemblage of fools and knaves . But you must be neither the one nor the other . I am giving you my name like Ariadne 's clue of thread to take with you into the labyrinth ; make no unworthy use of it , " she said , with a queenly glance and curve of her throat ; " give it back to me unsullied . And now , go ; leave me . We women also have our battles to fight . " " And if you should ever need some one who would gladly set a match to a train for you---- " " Well ? " she asked . He tapped his heart , smiled in answer to his cousin 's smile , and went . It was five o'clock , and Eugene was hungry ; he was afraid lest he should not be in time for dinner , a misgiving which made him feel that it was pleasant to be borne so quickly across Paris . This sensation of physical comfort left his mind free to grapple with the thoughts that assailed him . A mortification usually send
 s a young man of his age into a furious rage ; he shakes his fist at society , and vows vengeance when his belief in himself is shaken . Just then Rastignac was overwhelmed by the words , " You have shut the Countess ' door against you . " " I shall call ! " he said to himself , " and if Mme . de Beauseant is right , if I never find her at home--I ... well , Mme . de Restaud shall meet me in every salon in Paris . I will learn to fence and have some pistol practice , and kill that Maxime of hers ! " " And money ? " cried an inward monitor . " How about money , where is that to come from ? " And all at once the wealth displayed in the Countess de Restaud 's drawing-room rose before his eyes . That was the luxury which Goriot 's daughter had loved too well , the gilding , the ostentatious splendor , the unintelligent luxury of the parvenu , the riotous extravagance of a courtesan . Then the attractive vision suddenly went under an eclipse as he remembered the stately grandeur of the H
 otel de Beauseant . As his fancy wandered among these lofty regions in the great world of Paris , innumerable dark thoughts gathered in his heart ; his ideas widened , and his conscience grew more elastic . He saw the world as it is ; saw how the rich lived beyond the jurisdiction of law and public opinion , and found in success the _ultima ratio mundi_ . " Vautrin is right , success is virtue ! " he said to himself . Arrived in the Rue Neuve-Sainte-Genevieve , he rushed up to his room for ten francs wherewith to satisfy the demands of the cabman , and went in to dinner . He glanced round the squalid room , saw the eighteen poverty-stricken creatures about to feed like cattle in their stalls , and the sight filled him with loathing . The transition was too sudden , and the contrast was so violent that it could not but act as a powerful stimulant ; his ambition developed and grew beyond all social bounds . On the one hand , he beheld a vision of social life in its most charming and r
 efined forms , of quick-pulsed youth , of fair , impassioned faces invested with all the charm of poetry , framed in a marvelous setting of luxury or art ; and , on the other hand , he saw a sombre picture , the miry verge beyond these faces , in which passion was extinct and nothing was left of the drama but the cords and pulleys and bare mechanism . Mme . de Beauseant 's counsels , the words uttered in anger by the forsaken lady , her petulant offer , came to his mind , and poverty was a ready expositor . Rastignac determined to open two parallel trenches so as to insure success ; he would be a learned doctor of law and a man of fashion . Clearly he was still a child ! Those two lines are asymptotes , and will never meet . " You are very dull , my lord Marquis , " said Vautrin , with one of the shrewd glances that seem to read the innermost secrets of another mind . " I am not in the humor to stand jokes from people who call me 'my lord Marquis , ' " answered Eugene . " A marquis 
 here in Paris , if he is not the veriest sham , ought to have a hundred thousand livres a year at least ; and a lodger in the Maison Vauquer is not exactly Fortune 's favorite . " Vautrin 's glance at Rastignac was half-paternal , half-contemptuous . " Puppy ! " it seemed to say ; " I should make one mouthful of him ! " Then he answered : " You are in a bad humor ; perhaps your visit to the beautiful Comtesse de Restaud was not a success . " " She has shut her door against me because I told her that her father dined at our table , " cried Rastignac . Glances were exchanged all round the room ; Father Goriot looked down . " You have sent some snuff into my eye , " he said to his neighbor , turning a little aside to rub his hand over his face . " Any one who molests Father Goriot will have henceforward to reckon with me , " said Eugene , looking at the old man 's neighbor ; " he is worth all the rest of us put together . --I am not speaking of the ladies , " he added , turning in the 
 direction of Mlle . Taillefer . Eugene 's remarks produced a sensation , and his tone silenced the dinner-table . Vautrin alone spoke . " If you are going to champion Father Goriot , and set up for his responsible editor into the bargain , you had need be a crack shot and know how to handle the foils , " he said , banteringly . " So I intend , " said Eugene . " Then you are taking the field to-day ? " " Perhaps , " Rastignac answered . " But I owe no account of myself to any one , especially as I do not try to find out what other people do of a night . " Vautrin looked askance at Rastignac . " If you do not mean to be deceived by the puppets , my boy , you must go behind and see the whole show , and not peep through holes in the curtain . That is enough , " he added , seeing that Eugene was about to fly into a passion . " We can have a little talk whenever you like . " There was a general feeling of gloom and constraint . Father Goriot was so deeply dejected by the student 's remark
  that he did not notice the change in the disposition of his fellow-lodgers , nor know that he had met with a champion capable of putting an end to the persecution . " Then , M. Goriot sitting there is the father of a countess , " said Mme . Vauquer in a low voice . " And of a baroness , " answered Rastignac . " That is about all he is capable of , " said Bianchon to Rastignac ; " I have taken a look at his head ; there is only one bump--the bump of Paternity ; he must be an _eternal father_ . " Eugene was too intent on his thoughts to laugh at Bianchon 's joke . He determined to profit by Mme . de Beauseant 's counsels , and was asking himself how he could obtain the necessary money . He grew grave . The wide savannas of the world stretched before his eyes ; all things lay before him , nothing was his . Dinner came to an end , the others went , and he was left in the dining-room . " So you have seen my daughter ? " Goriot spoke tremulously , and the sound of his voice broke in upon
  Eugene 's dreams . The young man took the elder 's hand , and looked at him with something like kindness in his eyes . " You are a good and noble man , " he said . " We will have some talk about your daughters by and by . " He rose without waiting for Goriot 's answer , and went to his room . There he wrote the following letter to his mother : -- " My Dear Mother , --Can you nourish your child from your breast again ? I am in a position to make a rapid fortune , but I want twelve hundred francs--I must have them at all costs . Say nothing about this to my father ; perhaps he might make objections , and unless I have the money , I may be led to put an end to myself , and so escape the clutches of despair . I will tell you everything when I see you . I will not begin to try to describe my present situation ; it would take volumes to put the whole story clearly and fully . I have not been gambling , my kind mother , I owe no one a penny ; but if you would preserve the life that you ga
 ve me , you must send me the sum I mention . As a matter of fact , I go to see the Vicomtesse de Beauseant ; she is using her influence for me ; I am obliged to go into society , and I have not a penny to lay out on clean gloves . I can manage to exist on bread and water , or go without food , if need be , but I cannot do without the tools with which they cultivate the vineyards in this country . I must resolutely make up my mind at once to make my way , or stick in the mire for the rest of my days. I know that all your hopes are set on me , and I want to realize them quickly . Sell some of your old jewelry , my kind mother ; I will give you other jewels very soon . I know enough of our affairs at home to know all that such a sacrifice means , and you must not think that I would lightly ask you to make it ; I should be a monster if I could . You must think of my entreaty as a cry forced from me by imperative necessity . Our whole future lies in the subsidy with which I must begin my
  first campaign , for life in Paris is one continual battle . If you cannot otherwise procure the whole of the money , and are forced to sell our aunt 's lace , tell her that I will send her some still handsomer , " and so forth . He wrote to ask each of his sisters for their savings--would they despoil themselves for him , and keep the sacrifice a secret from the family ? To his request he knew that they would not fail to respond gladly , and he added to it an appeal to their delicacy by touching the chord of honor that vibrates so loudly in young and high-strung natures . Yet when he had written the letters , he could not help feeling misgivings in spite of his youthful ambition ; his heart beat fast , and he trembled . He knew the spotless nobleness of the lives buried away in the lonely manor house ; he knew what trouble and what joy his request would cause his sisters , and how happy they would be as they talked at the bottom of the orchard of that dear brother of theirs in Par
 is . Visions rose before his eyes ; a sudden strong light revealed his sisters secretly counting over their little store , devising some girlish stratagem by which the money could be sent to him _incognito_ , essaying , for the first time in their lives , a piece of deceit that reached the sublime in its unselfishness . " A sister 's heart is a diamond for purity , a deep sea of tenderness ! " he said to himself . He felt ashamed of those letters . What power there must be in the petitions put up by such hearts ; how pure the fervor that bears their souls to Heaven in prayer ! What exquisite joy they would find in self-sacrifice ! What a pang for his mother 's heart if she could not send him all that he asked for ! And this noble affection , these sacrifices made at such terrible cost , were to serve as the ladder by which he meant to climb to Delphine de Nucingen . A few tears , like the last grains of incense flung upon the sacred alter fire of the hearth , fell from his eyes . He
  walked up and down , and despair mingled with his emotion . Father Goriot saw him through the half-open door . " What is the matter , sir ? " he asked from the threshold . " Ah ! my good neighbor , I am as much a son and brother as you are a father . You do well to fear for the Comtesse Anastasie ; there is one M. Maxime de Trailles , who will be her ruin . " Father Goriot withdrew , stammering some words , but Eugene failed to catch their meaning . The next morning Rastignac went out to post his letters . Up to the last moment he wavered and doubted , but he ended by flinging them into the box . " I shall succeed ! " he said to himself . So says the gambler ; so says the great captain ; but the three words that have been the salvation of some few , have been the ruin of many more . A few days after this Eugene called at Mme . de Restaud 's house ; she was not at home . Three times he tried the experiment , and three times he found her doors closed against him , though he was caref
 ul to choose an hour when M. de Trailles was not there . The Vicomtesse was right . The student studied no longer . He put in an appearance at lectures simply to answer to his name , and after thus attesting his presence , departed forthwith . He had been through a reasoning process familiar to most students . He had seen the advisability of deferring his studies to the last moment before going up for his examinations ; he made up his mind to cram his second and third years ' work into the third year , when he meant to begin to work in earnest , and to complete his studies in law with one great effort . In the meantime he had fifteen months in which to navigate the ocean of Paris , to spread the nets and set the lines that would bring him a protectress and a fortune . Twice during that week he saw Mme . de Beauseant ; he did not go to her house until he had seen the Marquis d'Ajuda drive away . Victory for yet a few more days was with the great lady , the most poetic figure in the F
 aubourg Saint-Germain ; and the marriage of the Marquis d'Ajuda-Pinto with Mlle . de Rochefide was postponed . The dread of losing her happiness filled those days with a fever of joy unknown before , but the end was only so much the nearer . The Marquis d'Ajuda and the Rochefides agreed that this quarrel and reconciliation was a very fortunate thing ; Mme . de Beauseant ( so they hoped ) would gradually become reconciled to the idea of the marriage , and in the end would be brought to sacrifice d'Ajuda 's morning visits to the exigencies of a man 's career , exigencies which she must have foreseen . In spite of the most solemn promises , daily renewed , M. d'Ajuda was playing a part , and the Vicomtesse was eager to be deceived . " Instead of taking a leap heroically from the window , she is falling headlong down the staircase , " said her most intimate friend , the Duchesse de Langeais . Yet this after-glow of happiness lasted long enough for the Vicomtesse to be of service to her 
 young cousin . She had a half-superstitious affection for him . Eugene had shown her sympathy and devotion at a crisis when a woman sees no pity , no real comfort in any eyes ; when if a man is ready with soothing flatteries , it is because he has an interested motive . Rastignac made up his mind that he must learn the whole of Goriot 's previous history ; he would come to his bearings before attempting to board the Maison de Nucingen . The results of his inquiries may be given briefly as follows : -- In the days before the Revolution , Jean-Joachim Goriot was simply a workman in the employ of a vermicelli maker . He was a skilful , thrifty workman , sufficiently enterprising to buy his master 's business when the latter fell a chance victim to the disturbances of 1789. Goriot established himself in the Rue de la Jussienne , close to the Corn Exchange . His plain good sense led him to accept the position of President of the Section , so as to secure for his business the protection o
 f those in power at that dangerous epoch . This prudent step had led to success ; the foundations of his fortune were laid in the time of the Scarcity ( real or artificial ) , when the price of grain of all kinds rose enormously in Paris . People used to fight for bread at the bakers ' doors ; while other persons went to the grocers ' shops and bought Italian paste foods without brawling over it . It was during this year that Goriot made the money , which , at a later time , was to give him all the advantage of the great capitalist over the small buyer ; he had , moreover , the usual luck of average ability ; his mediocrity was the salvation of him . He excited no one 's envy , it was not even suspected that he was rich till the peril of being rich was over , and all his intelligence was concentrated , not on political , but on commercial speculations . Goriot was an authority second to none on all questions relating to corn , flour , and " middlings " ; and the production , storage
  , and quality of grain . He could estimate the yield of the harvest , and foresee market prices ; he bought his cereals in Sicily , and imported Russian wheat . Any one who had heard him hold forth on the regulations that control the importation and exportation of grain , who had seen his grasp of the subject , his clear insight into the principles involved , his appreciation of weak points in the way that the system worked , would have thought that here was the stuff of which a minister is made . Patient , active , and persevering , energetic and prompt in action , he surveyed his business horizon with an eagle eye . Nothing there took him by surprise ; he foresaw all things , knew all that was happening , and kept his own counsel ; he was a diplomatist in his quick comprehension of a situation ; and in the routine of business he was as patient and plodding as a soldier on the march . But beyond this business horizon he could not see . He used to spend his hours of leisure on the 
 threshold of his shop , leaning against the framework of the door . Take him from his dark little counting-house , and he became once more the rough , slow-witted workman , a man who cannot understand a piece of reasoning , who is indifferent to all intellectual pleasures , and falls asleep at the play , a Parisian Dolibom in short , against whose stupidity other minds are powerless . Natures of this kind are nearly all alike ; in almost all of them you will find some hidden depth of sublime affection . Two all-absorbing affections filled the vermicelli maker 's heart to the exclusion of every other feeling ; into them he seemed to put all the forces of his nature , as he put the whole power of his brain into the corn trade . He had regarded his wife , the only daughter of a rich farmer of La Brie , with a devout admiration ; his love for her had been boundless . Goriot had felt the charm of a lovely and sensitive nature , which , in its delicate strength , was the very opposite of 
 his own . Is there any instinct more deeply implanted in the heart of man than the pride of protection , a protection which is constantly exerted for a fragile and defenceless creature ? Join love thereto , the warmth of gratitude that all generous souls feel for the source of their pleasures , and you have the explanation of many strange incongruities in human nature . After seven years of unclouded happiness , Goriot lost his wife . It was very unfortunate for him . She was beginning to gain an ascendency over him in other ways ; possibly she might have brought that barren soil under cultivation , she might have widened his ideas and given other directions to his thoughts . But when she was dead , the instinct of fatherhood developed in him till it almost became a mania . All the affection balked by death seemed to turn to his daughters , and he found full satisfaction for his heart in loving them . More or less brilliant proposals were made to him from time to time ; wealthy merc
 hants or farmers with daughters vied with each other in offering inducements to him to marry again ; but he determined to remain a widower . His father-in-law , the only man for whom he felt a decided friendship , gave out that Goriot had made a vow to be faithful to his wife 's memory . The frequenters of the Corn Exchange , who could not comprehend this sublime piece of folly , joked about it among themselves , and found a ridiculous nickname for him . One of them ventured ( after a glass over a bargain ) to call him by it , and a blow from the vermicelli maker 's fist sent him headlong into a gutter in the Rue Oblin . He could think of nothing else when his children were concerned ; his love for them made him fidgety and anxious ; and this was so well known , that one day a competitor , who wished to get rid of him to secure the field to himself , told Goriot that Delphine had just been knocked down by a cab . The vermicelli maker turned ghastly pale , left the Exchange at once ,
  and did not return for several days afterwards ; he was ill in consequence of the shock and the subsequent relief on discovering that it was a false alarm . This time , however , the offender did not escape with a bruised shoulder ; at a critical moment in the man 's affairs , Goriot drove him into bankruptcy , and forced him to disappear from the Corn Exchange . As might have been expected , the two girls were spoiled . With an income of sixty thousand francs , Goriot scarcely spent twelve hundred on himself , and found all his happiness in satisfying the whims of the two girls . The best masters were engaged , that Anastasie and Delphine might be endowed with all the accomplishments which distinguish a good education . They had a chaperon--luckily for them , she was a woman who had good sense and good taste ; --they learned to ride ; they had a carriage for their use ; they lived as the mistress of a rich old lord might live ; they had only to express a wish , their father would 
 hasten to give them their most extravagant desires , and asked nothing of them in return but a kiss . Goriot had raised the two girls to the level of the angels ; and , quite naturally , he himself was left beneath them . Poor man ! he loved them even for the pain that they gave him . When the girls were old enough to be married , they were left free to choose for themselves . Each had half her father 's fortune as her dowry ; and when the Comte de Restaud came to woo Anastasie for her beauty , her social aspirations led her to leave her father 's house for a more exalted sphere . Delphine wished for money ; she married Nucingen , a banker of German extraction , who became a Baron of the Holy Roman Empire . Goriot remained a vermicelli maker as before . His daughters and his sons-in-law began to demur ; they did not like to see him still engaged in trade , though his whole life was bound up with his business . For five years he stood out against their entreaties , then he yielded , 
 and consented to retire on the amount realized by the sale of his business and the savings of the last few years . It was this capital that Mme . Vauquer , in the early days of his residence with her , had calculated would bring in eight or ten thousand livres in a year . He had taken refuge in her lodging-house , driven there by despair when he knew that his daughters were compelled by their husbands not only to refuse to receive him as an inmate in their houses , but even to see him no more except in private . This was all the information which Rastignac gained from a M. Muret who had purchased Goriot 's business , information which confirmed the Duchesse de Langeais ' suppositions , and herewith the preliminary explanation of this obscure but terrible Parisian tragedy comes to an end . 
\ No newline at end of file


[48/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/jsmlearning/JSMLearnerOnLatticeWithDeduction.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/jsmlearning/JSMLearnerOnLatticeWithDeduction.java b/opennlp-similarity/src/main/java/opennlp/tools/jsmlearning/JSMLearnerOnLatticeWithDeduction.java
new file mode 100644
index 0000000..cea7187
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/jsmlearning/JSMLearnerOnLatticeWithDeduction.java
@@ -0,0 +1,243 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.jsmlearning;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.commons.collections.ListUtils;
+
+import edu.stanford.nlp.util.StringUtils;
+import opennlp.tools.parse_thicket.pattern_structure.LinguisticPatternStructure;
+
+import opennlp.tools.similarity.apps.utils.Pair;
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+
+public class JSMLearnerOnLatticeWithDeduction extends JSMLearnerOnLatticeBase{
+	List<JSMDecision> accumulatedJSMResults = new ArrayList<JSMDecision>();
+
+
+
+	public JSMDecision buildLearningModel(List<String> posTexts, List<String> negTexts, 
+			String unknown, String[] separationKeywords){
+		psPos = new LinguisticPatternStructure(0,0); psNeg = new LinguisticPatternStructure(0,0);
+		if (separationKeywords!=null){ // re-sort by occurrence of separation keyword
+			Pair<List<String>, List<String>> pair = reGroupByOccurrenceOfSeparationKeyword(posTexts, negTexts, separationKeywords );
+			posTexts = pair.getFirst(); negTexts = 	pair.getSecond();
+		}
+
+		List<List<List<ParseTreeChunk>>> lingRepsPos = new ArrayList<List<List<ParseTreeChunk>>>(),
+				lingRepsNeg = new ArrayList<List<List<ParseTreeChunk>>>();
+		for(String text: posTexts)
+			lingRepsPos.add(chunk_maker.formGroupedPhrasesFromChunksForPara(text));
+
+		for(String text: negTexts)
+			lingRepsNeg.add(chunk_maker.formGroupedPhrasesFromChunksForPara(text));
+
+		LinkedHashSet<Integer> obj = null;
+		int i=0;
+		for(List<List<ParseTreeChunk>> chunk: lingRepsPos){
+			obj = new LinkedHashSet<Integer>();
+			obj.add(i);
+			psPos.AddIntent(chunk, obj, 0);
+			i++;
+		}
+		i=0;
+		for(List<List<ParseTreeChunk>> chunk: lingRepsNeg){
+			obj = new LinkedHashSet<Integer>();
+			obj.add(i);
+			psNeg.AddIntent(chunk, obj, 0);
+			i++;
+		}
+
+		List<List<ParseTreeChunk>> chunksUnknown = chunk_maker.formGroupedPhrasesFromChunksForPara(unknown);
+		List<List<List<ParseTreeChunk>>> posIntersections = new ArrayList<List<List<ParseTreeChunk>>>(), 
+				negIntersections = new ArrayList<List<List<ParseTreeChunk>>>();
+		List<List<ParseTreeChunk>> intersection = null;
+		for(int iConcept = 0; iConcept<psPos.conceptList.size(); iConcept++){
+			if (psPos.conceptList.get(iConcept).intent!=null && psPos.conceptList.get(iConcept).intent.size()>0){
+				intersection =  computeIntersectionWithIntentExtendedByDeduction(psPos, iConcept, chunksUnknown);
+				if (reduceList(intersection).size()>0)
+					posIntersections.add(reduceList(intersection));
+			}
+		}
+		for(int iConcept = 0; iConcept<psNeg.conceptList.size(); iConcept++){
+			if (psNeg.conceptList.get(iConcept).intent!=null && psNeg.conceptList.get(iConcept).intent.size()>0){				
+				intersection = computeIntersectionWithIntentExtendedByDeduction(psNeg, iConcept, chunksUnknown);
+				if (reduceList(intersection).size()>0)
+					negIntersections.add(reduceList(intersection));
+			}
+		}
+
+		Pair<List<List<List<ParseTreeChunk>>>, List<List<List<ParseTreeChunk>>>> pair = 
+				removeInconsistenciesFromPosNegIntersections( posIntersections, 
+						negIntersections);
+
+		posIntersections = pair.getFirst();
+		negIntersections = pair.getSecond();
+
+		List<List<List<ParseTreeChunk>>> posIntersectionsUnderNeg = new ArrayList<List<List<ParseTreeChunk>>>(), 
+				negIntersectionsUnderPos = new ArrayList<List<List<ParseTreeChunk>>>();
+
+		for(int iConcept = 0; iConcept<psNeg.conceptList.size(); iConcept++){
+			for(int iConceptJ = 0; iConceptJ<negIntersections.size(); iConceptJ++){
+				intersection = md
+						.matchTwoSentencesGroupedChunksDeterministic(psNeg.conceptList.get(iConcept).intent, negIntersections.get(iConceptJ));
+				if (reduceList(intersection).size()>0)
+					posIntersectionsUnderNeg.add(reduceList(intersection));
+			}
+		}
+
+		for(int iConcept = 0; iConcept<psPos.conceptList.size(); iConcept++){
+			for(int iConceptJ = 0; iConceptJ<posIntersections.size(); iConceptJ++){
+				intersection = md
+						.matchTwoSentencesGroupedChunksDeterministic(psPos.conceptList.get(iConcept).intent, posIntersections.get(iConceptJ));
+				if (reduceList(intersection).size()>0)
+					negIntersectionsUnderPos.add(reduceList(intersection));
+			}
+		}
+
+		List<ParseTreeChunk>posIntersectionsUnderNegLst = flattenParseTreeChunkLst(posIntersectionsUnderNeg);
+		List<ParseTreeChunk>negIntersectionsUnderPosLst=flattenParseTreeChunkLst(negIntersectionsUnderPos);
+
+		posIntersectionsUnderNegLst = subtract(posIntersectionsUnderNegLst, negIntersectionsUnderPosLst);
+		negIntersectionsUnderPosLst= subtract(negIntersectionsUnderPosLst, posIntersectionsUnderNegLst);
+
+		System.out.println("Pos - neg inters = "+posIntersectionsUnderNegLst);
+		System.out.println("Neg - pos inters = "+negIntersectionsUnderPosLst);
+
+		Boolean bPositiveClass = (float)posIntersectionsUnderNegLst.size()/(float)negIntersectionsUnderPosLst.size() > 1f;
+
+		JSMDecision decision = new JSMDecision("keywordClassName", bPositiveClass, 
+				posIntersections , negIntersections, 
+				posIntersectionsUnderNeg,
+				negIntersectionsUnderPos, separationKeywords);
+
+		accumulatedJSMResults.add(decision);
+
+		return decision;
+
+	}
+
+	private List<List<ParseTreeChunk>> computeIntersectionWithIntentExtendedByDeduction(
+			LinguisticPatternStructure psPos, int iConcept,
+			List<List<ParseTreeChunk>> chunksUnknown) {
+		
+		List<List<ParseTreeChunk>> intent = psPos.conceptList.get(iConcept).intent, 
+				intentExtendedByDeduction = new ArrayList<List<ParseTreeChunk>>();
+		
+	
+		for(  List<ParseTreeChunk> group: intent){
+			List<ParseTreeChunk> newGroup = new ArrayList<ParseTreeChunk>();
+			for(ParseTreeChunk ch: group){
+					newGroup.add(ch);
+					List<String> lemmas = ch.getLemmas();
+					List<List<List<ParseTreeChunk>>> clausesBodiesToAdd = findClausesForListOfLemmas(lemmas);
+					if (clausesBodiesToAdd!=null && clausesBodiesToAdd.size()>0)
+						intentExtendedByDeduction .add(flattenParseTreeChunkLst(clausesBodiesToAdd));
+			}
+			intentExtendedByDeduction .add(newGroup);
+		} 
+		 return md
+			.matchTwoSentencesGroupedChunksDeterministic(intentExtendedByDeduction, chunksUnknown);
+		
+	}
+    
+	// for list of words in a phrase, identify if it includes a separation word/multiword and get respective clause body 
+	//(to add to existing intent by the calling func)
+	private List<List<List<ParseTreeChunk>>> findClausesForListOfLemmas(
+			List<String> lemmas) {
+		for(String lemma: lemmas){
+			for(JSMDecision dec:  accumulatedJSMResults ){
+				String[] sepKeywords = dec.getSeparationKeywords();
+				// if all separation keywords occur in this phrase
+				if (ListUtils.intersection(lemmas, Arrays.asList(sepKeywords)).size() == sepKeywords.length){
+					return dec.getPosHypotheses();
+				}
+			}
+		}
+		return null;
+	}
+
+	public Pair<List<String>, List<String>>  reGroupByOccurrenceOfSeparationKeyword(List<String> posTexts, List<String> negTexts, String[] keywords){
+		List<String> posTextsNew = new ArrayList<String>(), negTextsNew = new ArrayList<String>();
+		for(String posText:posTexts){
+			Boolean multiwordOccurs = true;
+			for(String keyword: keywords){
+				if (!StringUtils.find(posText, keyword))
+					multiwordOccurs = false;
+				break;
+			}
+			if (multiwordOccurs)
+				posTextsNew.add(posText);
+			else
+				negTextsNew.add(posText);
+		}
+		for(String negText:negTexts){
+			Boolean multiwordOccurs = true;
+			for(String keyword: keywords){
+				if (!StringUtils.find(negText, keyword))
+					multiwordOccurs = false;
+				break;
+			}
+			if (multiwordOccurs)
+				posTextsNew.add(negText);
+			else
+				negTextsNew.add(negText);
+		}
+
+
+		return new Pair<List<String>, List<String>>(posTextsNew , negTextsNew);
+	}
+
+	
+
+	public static void main (String[] args) {
+
+		String[] posArr = new String[] {"I rent an office space. This office is for my business. I can deduct office rental expense from my business profit to calculate net income. ",
+				"To run my business, I have to rent an office. The net business profit is calculated as follows. Rental expense needs to be subtracted from revenue. ",
+				"To store goods for my retail business I rent some space. When I calculate the net income, I take revenue and subtract business expenses such as office rent. ",
+		"I rent some space for my business. To calculate my net income, I subtract from revenue my rental business expense."};
+
+		String[] negArr = new String[] {"I rent out a first floor unit of my house to a travel business. I need to add the rental income to my profit. However, when I repair my house, I can deduct the repair expense from my rental income. ",
+				"I receive rental income from my office. I have to claim it as a profit in my tax forms. I need to add my rental income to my profits, but subtract rental expenses such as repair from it. ",
+				"I advertised my property as a business rental. Advertisement and repair expenses can be subtracted from the rental income. Remaining rental income needs to be added to my profit and be reported as taxable profit. ",			
+		"I showed  my property to a business owner to rent. Expenses on my time spent on advertisement are subtracted from the rental income. My rental profits are added to my taxable income.  "};	
+
+		String unknown = "I do not want to rent anything to anyone. I just want to rent a space for myself. I neither calculate deduction of individual or business tax. I subtract my tax from my income";
+		JSMLearnerOnLatticeWithDeduction jsm = new JSMLearnerOnLatticeWithDeduction();
+		JSMDecision dec1 =  // may be determined by 'subtract'
+				jsm.buildLearningModel(Arrays.asList(posArr), Arrays.asList(negArr), unknown , new String[]{"subtract"});
+		JSMDecision dec2 = // may be determined by ...
+				jsm.buildLearningModel(Arrays.asList(posArr), Arrays.asList(negArr), unknown , new String[]{"business"});
+		JSMDecision dec3 = // may be determined by ...
+				jsm.buildLearningModel(Arrays.asList(posArr), Arrays.asList(negArr), unknown , new String[]{"property"});
+		// Finally, do prediction
+		JSMDecision dec = // may be determined by ...
+				jsm.buildLearningModel(Arrays.asList(posArr), Arrays.asList(negArr), unknown , new String[]{"property"});
+		
+		
+		
+
+
+	}
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/ParseCorefBuilderWithNER.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/ParseCorefBuilderWithNER.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/ParseCorefBuilderWithNER.java
new file mode 100644
index 0000000..3841592
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/ParseCorefBuilderWithNER.java
@@ -0,0 +1,156 @@
+package opennlp.tools.parse_thicket;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import edu.stanford.nlp.dcoref.CorefChain;
+import edu.stanford.nlp.dcoref.CorefCoreAnnotations;
+import edu.stanford.nlp.dcoref.CorefChain.CorefMention;
+import edu.stanford.nlp.ie.AbstractSequenceClassifier;
+import edu.stanford.nlp.ie.crf.CRFClassifier;
+import edu.stanford.nlp.ling.CoreAnnotations;
+import edu.stanford.nlp.ling.CoreLabel;
+import edu.stanford.nlp.ling.CoreAnnotations.NamedEntityTagAnnotation;
+import edu.stanford.nlp.ling.CoreAnnotations.PartOfSpeechAnnotation;
+import edu.stanford.nlp.ling.CoreAnnotations.TextAnnotation;
+import edu.stanford.nlp.ling.CoreAnnotations.TokensAnnotation;
+import edu.stanford.nlp.neural.rnn.RNNCoreAnnotations;
+import edu.stanford.nlp.pipeline.Annotation;
+import edu.stanford.nlp.sentiment.SentimentCoreAnnotations.SentimentAnnotatedTree;
+import edu.stanford.nlp.trees.Tree;
+import edu.stanford.nlp.trees.TreeCoreAnnotations;
+import edu.stanford.nlp.util.CoreMap;
+import opennlp.tools.parse_thicket.ArcType;
+import opennlp.tools.parse_thicket.Pair;
+import opennlp.tools.parse_thicket.ParseCorefsBuilder;
+import opennlp.tools.parse_thicket.ParseThicket;
+import opennlp.tools.parse_thicket.ParseTreeNode;
+import opennlp.tools.parse_thicket.WordWordInterSentenceRelationArc;
+
+public class ParseCorefBuilderWithNER extends ParseCorefsBuilder {
+
+	private static ParseCorefBuilderWithNER instanceNER;
+
+	public synchronized static ParseCorefBuilderWithNER  getInstance() {
+		if (instanceNER == null)
+			instanceNER = new ParseCorefBuilderWithNER ();
+
+		return instanceNER;
+	}
+
+
+	AbstractSequenceClassifier<CoreLabel> classifier = null;
+
+	ParseCorefBuilderWithNER() {
+		super();
+		classifier = CRFClassifier.getDefaultClassifier();
+	}
+
+	public ParseThicket buildParseThicket(String text){
+		List<Tree> ptTrees = new ArrayList<Tree>();
+		// all numbering from 1, not 0
+		List<WordWordInterSentenceRelationArc> arcs = new ArrayList<WordWordInterSentenceRelationArc>();
+		List<List<ParseTreeNode>> nodesThicket = new ArrayList<List<ParseTreeNode>>();
+		List<Float> sentimentProfile = new ArrayList<Float>();
+
+		annotation = new Annotation(text);
+		try {
+			pipeline.annotate(annotation);
+			List<CoreMap> sentences = annotation.get(CoreAnnotations.SentencesAnnotation.class);
+			List<List<CoreLabel>> nerClassesText = classifier.classify(text);
+			
+
+			int nSent = 0;
+			if (sentences != null && sentences.size() > 0) 
+				for(CoreMap sentence: sentences){
+					List<ParseTreeNode> nodes = new ArrayList<ParseTreeNode>();
+					if (nSent>=nerClassesText.size())
+						break;
+					List<CoreLabel> nerClassesSent = nerClassesText .get(nSent);
+
+					// traversing the words in the current sentence
+					// a CoreLabel is a CoreMap with additional token-specific methods
+					Class<TokensAnnotation> tokenAnn = TokensAnnotation.class;
+					List<CoreLabel> coreLabelList = sentence.get(tokenAnn);
+					int count=1;
+					for (CoreLabel token: coreLabelList ) {
+						if (count-1>=nerClassesSent.size())
+							break;
+						CoreLabel classNerWord = 	nerClassesSent .get(count-1);
+						// this is the text of the token
+						String lemma = token.get(TextAnnotation.class);
+						// this is the POS tag of the token
+						String pos = token.get(PartOfSpeechAnnotation.class);
+						// this is the NER label of the token
+						String ne = token.get(NamedEntityTagAnnotation.class);     
+
+
+						ParseTreeNode p = new ParseTreeNode(lemma, pos, ne, count);
+						String ner = classNerWord .get(CoreAnnotations.AnswerAnnotation.class);
+						if (!ner.equals("O")){
+							Map<String, Object> nerMap = new HashMap<String, Object>();
+							nerMap.put("ner", ner);
+							p.setAttributes(nerMap);
+						}
+						nodes.add(p);
+						count++;
+					}	
+					nSent++;
+					nodesThicket.add(nodes);
+					Tree tree = sentence.get(TreeCoreAnnotations.TreeAnnotation.class);
+					// now sentiment for given sentence
+					Tree sentimentTree = sentence.get(SentimentAnnotatedTree.class);
+					float sentiment = RNNCoreAnnotations.getPredictedClass(sentimentTree);
+					sentimentProfile.add(sentiment);
+					ptTrees.add(tree);
+				}
+		} catch (Exception e) {
+			e.printStackTrace();
+		}
+
+
+		// now coreferences
+		Map<Integer, CorefChain> corefs = annotation.get(CorefCoreAnnotations.CorefChainAnnotation.class);
+		List<CorefChain> chains = new ArrayList<CorefChain>(corefs.values());
+		for(CorefChain c: chains){
+			//System.out.println(c);
+			List<CorefMention> mentions = c.getMentionsInTextualOrder();
+			//System.out.println(mentions);
+			if (mentions.size()>1)
+				for(int i=0; i<mentions.size(); i++){
+					for(int j=i+1; j<mentions.size(); j++){
+						CorefMention mi = mentions.get(i), mj=mentions.get(j);
+
+
+						int niSentence = mi.position.get(0);
+						int niWord = mi.startIndex;
+						int njSentence = mj.position.get(0);
+						int njWord = mj.startIndex;
+
+						ArcType arcType = new ArcType("coref-", mj.mentionType+"-"+mj.animacy, 0, 0);
+
+						WordWordInterSentenceRelationArc arc = 
+								new WordWordInterSentenceRelationArc(new Pair<Integer, Integer>(niSentence,niWord), 
+										new Pair<Integer, Integer>(njSentence,njWord), mi.mentionSpan, mj.mentionSpan, 
+										arcType);
+						arcs.add(arc);
+					}
+				}
+		}
+		List<WordWordInterSentenceRelationArc> arcsCA = buildCAarcs(nodesThicket);
+		arcs.addAll(arcsCA);
+
+		ParseThicket result = new ParseThicket(ptTrees, arcs);
+		result.setSentimentProfile(sentimentProfile);
+		result.setNodesThicket(nodesThicket);
+		return result;
+	}
+
+	public static void main(String[] args){
+		new ParseCorefBuilderWithNER ().buildParseThicket("No one knows yet what General Prayuth's real intentions are. He has good reason to worry about resistance. "
+				+ "The pro-government Red-Shirt movement is far better organised than eight years ago, and could still be financed by former Prime Minister Thaksin Shinawatra's deep pockets.");
+	}
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/VerbNetProcessor.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/VerbNetProcessor.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/VerbNetProcessor.java
new file mode 100644
index 0000000..166d4ff
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/VerbNetProcessor.java
@@ -0,0 +1,267 @@
+package opennlp.tools.parse_thicket;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import edu.mit.jverbnet.data.Frame;
+import edu.mit.jverbnet.data.Frame.FrameBuilder;
+import edu.mit.jverbnet.data.FrameType;
+import edu.mit.jverbnet.data.IFrame;
+import edu.mit.jverbnet.data.IMember;
+import edu.mit.jverbnet.data.IThematicRole;
+import edu.mit.jverbnet.data.IVerbClass;
+import edu.mit.jverbnet.data.IWordnetKey;
+import edu.mit.jverbnet.data.VerbClass;
+import edu.mit.jverbnet.index.IVerbIndex;
+import edu.mit.jverbnet.index.VerbIndex;
+
+public class VerbNetProcessor implements IGeneralizer<Map<String, List<String>>> {
+
+	static VerbNetProcessor instance;
+	static private String pathToVerbnet = null; //new File( "." ).getCanonicalPath()+"/src/test/resources";
+	public static VerbNetProcessor getInstance(String resourceDir) {
+		if (resourceDir==null)
+			try {
+				resourceDir = new File( "." ).getCanonicalPath()+"/src/test/resources";
+			} catch (IOException e) {
+				e.printStackTrace();
+			}
+		pathToVerbnet = resourceDir + "/new_vn";
+		if (instance == null)
+			instance = new VerbNetProcessor();
+
+		return instance;
+	}	
+
+	IVerbIndex index = null;
+
+	private VerbNetProcessor() {
+
+		try {
+			URL url = new URL ("file", null , pathToVerbnet ) ;
+			index = new VerbIndex ( url ) ;
+
+			index . open () ;
+
+		} catch (MalformedURLException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		} catch (IOException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		}
+	}
+
+
+	public IVerbClass getVerbNetForAVerb_____New(String verb){
+		Iterator<IVerbClass> iter = index.iterator();
+		while(iter.hasNext()){
+			IVerbClass v = iter.next();
+
+			if (v.getID().startsWith(verb))
+				return v;
+		}
+		iter = index.iterator();
+		while(iter.hasNext()){
+			IVerbClass v = iter.next();
+			if (!v.getMembers().isEmpty()){
+				for(IMember m: v.getMembers()) {
+					if (m.getName().equals(verb)){
+						return v;
+					}
+				}
+			}
+		}
+		return null;
+	}
+
+
+
+	public IVerbClass getVerbNetForAVerb(String verb){
+		Iterator<IVerbClass> iter = index.iterator();
+		while(iter.hasNext()){
+			IVerbClass v = iter.next();
+
+			if (v.getID().startsWith(verb))
+				return v;
+
+			if (!v.getMembers().isEmpty() && v.getMembers().get(0).getName().equals(verb)){
+				return v;
+			}
+		}
+		return null;
+	}
+
+	public List<Map<String, List<String>>> generalize(Object o1, Object o2) {
+		IVerbClass v1, v2;
+		if ((o1 instanceof String) && (o2 instanceof String)){
+			v1 = getVerbNetForAVerb((String) o1);
+			v2 = getVerbNetForAVerb((String) o2);		
+			return generalize(v1, v2);
+		} else
+
+			v1 = (IVerbClass)o1;
+		v2 = (IVerbClass)o2;
+		List<Map<String, List<String>>> resList = new ArrayList<Map<String, List<String>>>();
+
+		if (v1 ==null || v2==null) // not found
+			return  resList;
+
+		// lists for results
+		List<String> roles = new ArrayList<String>();
+
+		List<IThematicRole> roles1 = v1.getThematicRoles(), roles2 = v2.getThematicRoles();
+		Map<String, List<String>> results = new HashMap<String, List<String>>();
+
+		for(int i=0; i< roles1.size()&& i< roles2.size(); i++){
+			if (roles1.get(i).getType().equals(roles2.get(i).getType())){
+				roles.add(roles1.get(i).getType().toString());
+			} else 
+				roles.add("*");
+		}
+
+		List<IFrame> frames1 = v1.getFrames(), frames2 = v2.getFrames();
+		List<String> patterns1 = new ArrayList<String>(), patterns2 = new ArrayList<String>();
+		for(int i=0; i< frames1.size(); i++){
+			patterns1.add(frames1.get(i).getPrimaryType().getID());
+		}
+		for(int i=0; i< frames2.size(); i++){
+			patterns2.add(frames2.get(i).getPrimaryType().getID());
+		}
+		patterns2.retainAll(patterns1);
+		results.put("phrStr", patterns2) ; 
+
+		List<String> patternsWord1 = new ArrayList<String>(), patternsWord2 = new ArrayList<String>();
+		for(int i=0; i< frames1.size(); i++){
+			try {
+				patternsWord1.add(frames1.get(i).getSecondaryType().getID());
+			} catch (Exception e) {
+				e.printStackTrace();
+			}
+		}
+		for(int i=0; i< frames2.size(); i++){
+			try {
+				if (frames2.get(i).getSecondaryType()!=null && frames2.get(i).getSecondaryType().getID()!=null)
+					patternsWord2.add(frames2.get(i).getSecondaryType().getID());
+			} catch (Exception e) {
+				// TODO Auto-generated catch block
+				e.printStackTrace();
+			}
+		}
+		patternsWord2.retainAll(patternsWord1);
+		results.put("phrDescr", patternsWord2) ; 
+
+		results.put("roles", roles);
+
+		resList.add(results);
+		return resList;
+	}
+
+	// takes a verb and forms its verbnet parameters 
+	// abandon  (leave-51.2 leave-51.2 ) (NP V NP.source ) (Transitivebasically, locative preposition drop of "from" ) (
+	public StringBuilder buildTreeRepresentationForTreeKernelLearning(String verb){
+		StringBuilder sb = new StringBuilder(1000);
+		IVerbClass v;
+		v = getVerbNetForAVerb(verb);
+		if (v==null) // for some reason this verb is not in the vocabulary
+			return null;
+		sb.append(verb + "  (" );
+		List<IThematicRole> roles = v.getThematicRoles();
+
+		for(int i=0; i< roles.size(); i++){
+			sb.append(roles.get(i).getVerbClass().getID().replace(".", "")+" ");
+		}
+		sb.append( ") (" );
+
+		List<IFrame> frames = v.getFrames();
+		for(int i=0; i< frames.size(); i++){
+			sb.append(//" ("+
+		frames.get(i).getPrimaryType().getID().replace(".", "-")+" ");
+		}
+		sb.append( ") (" );
+		for(int i=0; i< frames.size(); i++){
+			sb.append(frames.get(i).getSecondaryType().getID().
+					replace(".", "").replace(",", " ").replace("\"", "-").replace("/", "-").replace("(","").replace(")","")+" ");
+		}
+		sb.append( ") " );
+
+		if (v.getParent()!=null && v.getParent().getThematicRoles()!=null){
+			sb.append( "(" );
+			for(int i=0; i<v.getParent().getThematicRoles().size(); i++){
+				sb.append(v.getParent().getThematicRoles().get(i).getType()+" ");
+			}
+			sb.append( ")" );
+		}
+		return sb;
+	}
+
+	public void testIndex () throws Exception {
+		Iterator<IVerbClass> iter = index.iterator();
+		while(iter.hasNext()){
+			IVerbClass v = iter.next();
+			System.out.println(v.getID() + " +> " + v.getFrames().get(0).getVerbClass().getID() + "  \n ===> " + v.getMembers().get(0).getName()  );
+			List<IThematicRole> roles = v.getThematicRoles();
+			for (IThematicRole r: roles){
+				System.out.println(r.getType());
+			}
+
+			List<IFrame> frames = v.getFrames();
+			for(IFrame f: frames){
+				try {
+					System.out.println(f.getPrimaryType().getID() + " => " + f.getXTag() + " >> "+ f.getSecondaryType().getID() +  " : " + f.getExamples().get(0));
+				} catch (Exception e) {
+					// TODO Auto-generated catch block
+					e.printStackTrace();
+				}
+			}
+		}
+		IVerbClass verb0 =  index.getVerb("hit-18.1");
+		// look up a verb class and print out some info
+		IVerbClass verb = index . getRootVerb ("hit-18.1") ;
+		IMember member = verb . getMembers () . get (0) ;
+		Set < IWordnetKey > keys = member . getWordnetTypes () . keySet () ;
+		IFrame frame = verb . getFrames () . get (0) ;
+		FrameType type = frame . getPrimaryType () ;
+		String example = frame . getExamples () . get (0) ;
+		System . out . println ("id: " + verb . getID () ) ;
+		System . out . println (" first wordnet keys : " + keys ) ;
+		System . out . println (" first frame type : " + type . getID () ) ;
+		System . out . println (" first example : " + example ) ;
+	}
+
+	public static void main(String[] args){
+		String resourceDir = new File(".").getAbsolutePath().replace("/.", "") + "/src/test/resources";
+		VerbNetProcessor proc = VerbNetProcessor.getInstance(resourceDir);
+		/*	try {
+				proc.testIndex();
+			} catch (Exception e) {
+				// TODO Auto-generated catch block
+				e.printStackTrace();
+			}
+		 */	
+
+		System.out.println(proc.buildTreeRepresentationForTreeKernelLearning("abandon"));
+		System.out.println(proc.buildTreeRepresentationForTreeKernelLearning("earn"));
+		
+		List res = proc.generalize("marry", "engage");
+		System.out.println (res);
+
+		res = proc.generalize("assume", "alert");
+		System.out.println (res);
+
+		res = proc.generalize("alert", "warn");
+		System.out.println (res);
+	}
+
+
+
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/external_rst/ExternalRSTImporter.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/external_rst/ExternalRSTImporter.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/external_rst/ExternalRSTImporter.java
new file mode 100644
index 0000000..955bcb8
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/external_rst/ExternalRSTImporter.java
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.parse_thicket.external_rst;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang.StringUtils;
+
+import edu.stanford.nlp.trees.Tree;
+import opennlp.tools.parse_thicket.ArcType;
+import opennlp.tools.parse_thicket.Pair;
+import opennlp.tools.parse_thicket.ParseThicket;
+import opennlp.tools.parse_thicket.ParseTreeNode;
+import opennlp.tools.parse_thicket.WordWordInterSentenceRelationArc;
+import opennlp.tools.parse_thicket.matching.Matcher;
+import opennlp.tools.parse_thicket.matching.PT2ThicketPhraseBuilder;
+import opennlp.tools.similarity.apps.utils.StringDistanceMeasurer;
+
+public class ExternalRSTImporter extends PT2ThicketPhraseBuilder{
+	private StringDistanceMeasurer strDistProc = new StringDistanceMeasurer ();
+	private String resourceDir = null; 
+	
+	public ExternalRSTImporter(){
+		 try {
+			resourceDir = new File( "." ).getCanonicalPath()+"/src/test/resources";
+		} catch (IOException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		}
+	}
+
+	public List<RstNode>  buildArrayOfRSTnodes(ParseThicket pt, String jotyDumpFileName){
+		String dump=null;
+		try {
+			dump = FileUtils.readFileToString(new File(jotyDumpFileName), Charset.defaultCharset().toString());
+		} catch (IOException e) {
+			e.printStackTrace();
+			return null;
+		}
+		List<RstNode> nodes = new ArrayList<RstNode>(); 
+		String[] lines = dump.split("\n");
+		int startOfDim = StringUtils.lastIndexOf(lines[0], " ");
+		String dimStr = lines[0].substring(startOfDim).replace(")", "").trim();
+		int dim = Integer.parseInt(dimStr);
+		Integer[][] rstArcsIndices = new Integer[dim][dim];
+		for(int i=1; i< lines.length; i++){
+			RstNode node = new RstNode(lines[i]);
+			nodes.add(node);
+		}
+		return nodes;
+	}
+
+	private Map<String, Integer > phraseRstIndex = new HashMap<String, Integer >();
+	private Map<Integer, List<ParseTreeNode> > rstIndexPhrase = new HashMap<Integer, List<ParseTreeNode> > ();
+
+	public List<WordWordInterSentenceRelationArc> buildRSTArcsFromRSTparser( List<RstNode> rstNodes,
+			List<WordWordInterSentenceRelationArc> arcs,
+			Map<Integer, List<List<ParseTreeNode>>> sentNumPhrasesMap, 
+			ParseThicket pt ) {
+		List<WordWordInterSentenceRelationArc> arcsRST = new ArrayList<WordWordInterSentenceRelationArc>();		
+
+		for(int nSentFrom=0; nSentFrom<pt.getSentences().size(); nSentFrom++){
+			for(int nSentTo=nSentFrom+1; nSentTo<pt.getSentences().size(); nSentTo++){
+
+				// label all phrases with EDU
+				List<List<ParseTreeNode>> phrasesFrom = sentNumPhrasesMap.get(nSentFrom);
+				for(List<ParseTreeNode> p: phrasesFrom ){
+					Integer rstIndex = findBestRstNodeTextForAPhrase(p, rstNodes);
+					if (rstIndex!=null){
+						phraseRstIndex.put(p.toString(), rstIndex );
+						rstIndexPhrase.put(rstIndex , p);
+					}
+				}
+				List<List<ParseTreeNode>> phrasesTo = sentNumPhrasesMap.get(nSentTo);
+				for(List<ParseTreeNode> p: phrasesTo ){
+					Integer rstIndex = findBestRstNodeTextForAPhrase(p, rstNodes);
+					if (rstIndex!=null){
+						phraseRstIndex.put(p.toString(), rstIndex );
+						rstIndexPhrase.put(rstIndex , p);
+					}
+				}
+			}
+		}	// for a pair of phrases, discover ^ in RST tree which connects these sentences
+		
+		for( int nSentFrom=0; nSentFrom<pt.getSentences().size(); nSentFrom++){
+			for(int nSentTo=nSentFrom+1; nSentTo<pt.getSentences().size(); nSentTo++){
+				System.out.println("Sent from # = "+nSentFrom + " -- " + "Sent to # = "+nSentTo);
+				
+				List<List<ParseTreeNode>> phrasesFrom = sentNumPhrasesMap.get(nSentFrom);
+				List<List<ParseTreeNode>> phrasesTo = sentNumPhrasesMap.get(nSentTo);
+				for(List<ParseTreeNode> vpFrom: phrasesFrom){
+					for(List<ParseTreeNode> vpTo: phrasesTo){
+						System.out.println("Computing arc between phrases "+ vpFrom + " => " + vpTo);
+						// get two RST nodes 
+						Integer rstNodeFrom = phraseRstIndex.get(vpFrom.toString());
+						Integer rstNodeTo = phraseRstIndex.get(vpTo.toString());
+						if (rstNodeFrom==null || rstNodeTo==null ||  rstNodeFrom >= rstNodeTo)
+							continue;
+						System.out.println("Finding RST path for phrases "+ vpFrom + "' and '"+vpTo);
+						System.out.println("Sent from # = "+nSentFrom + " -- " + "Sent to # = "+nSentTo);
+
+						Integer commonAncestorIndex = findCommonAncestor(rstNodeFrom , rstNodeTo, rstNodes);
+						if (commonAncestorIndex!=null){
+							// and figure out if they can be properly connected by an arc, by navigating RST tree
+							ArcType arcType = new ArcType("rst", rstNodes.get(rstNodeTo).getRel2par(), 0, 0);
+							WordWordInterSentenceRelationArc arcRST = 
+									new WordWordInterSentenceRelationArc(new Pair<Integer, Integer>(nSentFrom, vpFrom.get(0).getId()), 
+											new Pair<Integer, Integer>(nSentTo,  vpFrom.get(0).getId()), "", "", arcType);
+							arcsRST.add(arcRST);
+
+						}
+					}
+				} 
+			}
+		}
+
+		return arcsRST;
+	}
+
+	private Integer findAncestorForRSTnode(Integer rstNodeFrom, List<RstNode> rstNodes){
+		RstNode initNode = rstNodes.get(rstNodeFrom);
+		if (initNode.level==null)
+			return null;
+
+		try {
+			int initLevel = initNode.level;
+			int iter=1; // start with moving one step up
+			while (rstNodeFrom-iter>=0) {
+				Integer currLevel= rstNodes.get(rstNodeFrom-iter).level;
+				if ( currLevel!=null && currLevel< initLevel ) // found ancestor
+					return rstNodeFrom-iter;
+				iter++;
+			}
+		} catch (Exception e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		}
+		return null;
+	}
+
+	private Integer findCommonAncestor(Integer rstNodeFrom, Integer rstNodeTo,
+			List<RstNode> rstNodes) {
+		List<Integer> ancestorsFrom = new ArrayList<Integer>() , ancestorsTo = new ArrayList<Integer>();
+		ancestorsFrom.add(rstNodeFrom);  ancestorsTo.add(rstNodeTo);
+		int curLevel = rstNodes.get(rstNodeTo).level;
+		Integer rstNodeFromCurrent = rstNodeFrom,  rstNodeToCurrent = rstNodeTo; 
+
+		while(curLevel>0){
+			if (rstNodeFromCurrent !=null) {
+				rstNodeFromCurrent = findAncestorForRSTnode(rstNodeFromCurrent, rstNodes);
+			}
+			if (rstNodeToCurrent != null){
+				rstNodeToCurrent = findAncestorForRSTnode(rstNodeToCurrent, rstNodes);
+			}
+			if (rstNodeFromCurrent !=null) {
+				ancestorsFrom.add(rstNodeFromCurrent);  
+			}
+			if (rstNodeToCurrent != null)
+				ancestorsTo.add(rstNodeToCurrent);
+
+			List<Integer> ancestorsFromCurr = new ArrayList<Integer>(ancestorsFrom);
+			ancestorsFromCurr.retainAll(ancestorsTo);
+			if (! ancestorsFromCurr.isEmpty()){
+				System.out.println("Found comm ancestor "+rstNodes.get(ancestorsFromCurr.get(0)).toString() + " id =  "+ancestorsFromCurr.get(0) + 
+						" for two RST nodes | id = "+rstNodeFrom + "'"+
+						rstNodes.get(rstNodeFrom).toString() + "' and | id = "+ rstNodeTo + "'"+ rstNodes.get(rstNodeTo).toString()+"'");
+				String rel2par =  rstNodes.get(ancestorsFromCurr.get(0)).rel2par;
+				// if common ancestor is trivial, return null and do not form a link
+				if (rel2par==null) // || rel2par.equals("span"))
+					return null;
+				else
+					return ancestorsFromCurr.get(0);
+			}
+			curLevel--;
+		}
+		return null;
+	}
+
+	private Integer findBestRstNodeTextForAPhrase(List<ParseTreeNode> ps,
+			List<RstNode> rstNodes) {
+		// firt get  the phrase string
+		String phraseStr="";
+		for(ParseTreeNode n: ps){
+			phraseStr+=" "+n.getWord();
+		}
+		phraseStr= phraseStr.trim();
+		if (phraseStr.length()<10){
+			return null;
+		}
+		// now look for closest EDU text from the list of all 
+		double rMin = -10000d; Integer index = -1;
+		int count =0;
+		for(RstNode r: rstNodes){
+			if (r.getText()==null || r.getText().length()<10){
+				count++;
+				continue;
+			}
+			double res =  strDistProc.measureStringDistanceNoStemming(phraseStr, r.getText());
+			if (res > rMin){
+				rMin=res;
+				index = count;
+			}
+			count++;
+		}
+		if (index==-1)
+			return null;
+		System.out.println("Found RST node "+ rstNodes.get(index) +" for phrase ="+phraseStr);
+		return index;
+	}
+
+	/* 
+	 * Building phrases takes a Parse Thicket and forms phrases for each sentence individually
+	 * Then based on built phrases and obtained arcs, it builds arcs for RST
+	 * Finally, based on all formed arcs, it extends phrases with thicket phrases
+	 */
+
+	public List<WordWordInterSentenceRelationArc> buildPT2ptPhrases(ParseThicket pt, String externalRSTresultFilename ) {
+		List<List<ParseTreeNode>> phrasesAllSent = new ArrayList<List<ParseTreeNode>> ();
+		Map<Integer, List<List<ParseTreeNode>>> sentNumPhrases = new HashMap<Integer, List<List<ParseTreeNode>>>();
+		// build regular phrases
+		for(int nSent=0; nSent<pt.getSentences().size(); nSent++){
+			List<ParseTreeNode> sentence = pt.getNodesThicket().get(nSent);
+			Tree ptree = pt.getSentences().get(nSent);
+			//ptree.pennPrint();
+			List<List<ParseTreeNode>> phrases = buildPT2ptPhrasesForASentence(ptree, sentence);
+			System.out.println(phrases);
+			phrasesAllSent.addAll(phrases);
+			sentNumPhrases.put(nSent, phrases);
+
+		}
+		// TODO: code to run joty suite
+		List<RstNode> rstNodes = new ExternalRSTImporter().buildArrayOfRSTnodes(null, resourceDir+externalRSTresultFilename );
+
+		// discover and add RST arcs
+		List<WordWordInterSentenceRelationArc> arcsRST = buildRSTArcsFromRSTparser(  rstNodes, null, sentNumPhrases, pt );
+		System.out.println(arcsRST);
+		return arcsRST;
+
+	}
+
+
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/external_rst/MatcherExternalRST.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/external_rst/MatcherExternalRST.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/external_rst/MatcherExternalRST.java
new file mode 100644
index 0000000..215ee35
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/external_rst/MatcherExternalRST.java
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.parse_thicket.external_rst;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import opennlp.tools.parse_thicket.IGeneralizer;
+import opennlp.tools.parse_thicket.ParseCorefBuilderWithNER;
+import opennlp.tools.parse_thicket.ParseCorefsBuilder;
+import opennlp.tools.parse_thicket.ParseThicket;
+import opennlp.tools.parse_thicket.ParseTreeNode;
+import opennlp.tools.parse_thicket.VerbNetProcessor;
+import opennlp.tools.parse_thicket.matching.GeneralizationResult;
+import opennlp.tools.parse_thicket.matching.Matcher;
+import opennlp.tools.parse_thicket.matching.PT2ThicketPhraseBuilder;
+import opennlp.tools.parse_thicket.matching.PhraseGroupGeneralizer;
+import opennlp.tools.textsimilarity.LemmaPair;
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+import opennlp.tools.textsimilarity.ParseTreeMatcherDeterministic;
+import opennlp.tools.textsimilarity.SentencePairMatchResult;
+import opennlp.tools.textsimilarity.chunker2matcher.ParserChunker2MatcherProcessor;
+
+public class MatcherExternalRST extends Matcher implements IGeneralizer<List<List<ParseTreeNode>>>{
+
+	ParseCorefBuilderWithNERandRST ptBuilderRST = new ParseCorefBuilderWithNERandRST();
+	PT2ThicketPhraseBuilder phraseBuilder = new PT2ThicketPhraseBuilder();
+
+	private String externRSTpath;
+
+	public List<List<ParseTreeChunk>> assessRelevance(String para1, String para2) {
+		// first build PTs for each text
+		ParseThicket pt1 = ptBuilderRST.buildParseThicket(para1);
+		ParseThicket pt2 = ptBuilderRST.buildParseThicket(para2);
+		// then build phrases and rst arcs
+		List<List<ParseTreeNode>> phrs1 = phraseBuilder.buildPT2ptPhrases(pt1);
+		List<List<ParseTreeNode>> phrs2 = phraseBuilder.buildPT2ptPhrases(pt2);
+		// group phrases by type
+		List<List<ParseTreeChunk>> sent1GrpLst = formGroupedPhrasesFromChunksForPara(phrs1), 
+				sent2GrpLst = formGroupedPhrasesFromChunksForPara(phrs2);
+
+		
+		List<List<ParseTreeChunk>> res = pgGen.generalize(sent1GrpLst, sent2GrpLst);
+				
+		return res;
+
+	}
+	
+	// this function is the main entry point into the PT builder if rst arcs are required
+		public ParseThicket buildParseThicketFromTextWithRST(String para){
+			ParseThicket pt = ptBuilderRST.buildParseThicket(para);
+			
+			List<List<ParseTreeNode>> phrs = phraseBuilder.buildPT2ptPhrases(pt);
+			if (pt!=null)
+				pt.setPhrases(phrs);
+			return pt;	
+		}
+	
+	public List<List<ParseTreeChunk>> assessRelevance(List<List<ParseTreeChunk>> para0, String para2) {
+		// first build PTs for each text
+	
+		ParseThicket pt2 = ptBuilder.buildParseThicket(para2);
+		// then build phrases and rst arcs
+		List<List<ParseTreeNode>> phrs2 = phraseBuilder.buildPT2ptPhrases(pt2);
+		// group phrases by type
+		List<List<ParseTreeChunk>> sent2GrpLst = formGroupedPhrasesFromChunksForPara(phrs2);
+		List<List<ParseTreeChunk>> res = pgGen.generalize(para0, sent2GrpLst);
+				
+		return res;
+
+	}
+	
+	
+	public static void main(String[] args){
+		//MatcherExternalRST m = new MatcherExternalRST();
+		
+		
+	}
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/external_rst/PT2ThicketPhraseBuilderExtrnlRST.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/external_rst/PT2ThicketPhraseBuilderExtrnlRST.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/external_rst/PT2ThicketPhraseBuilderExtrnlRST.java
new file mode 100644
index 0000000..b915543
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/external_rst/PT2ThicketPhraseBuilderExtrnlRST.java
@@ -0,0 +1,115 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.parse_thicket.external_rst;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.logging.Logger;
+
+import opennlp.tools.parse_thicket.ParseThicket;
+import opennlp.tools.parse_thicket.ParseTreeNode;
+import opennlp.tools.parse_thicket.WordWordInterSentenceRelationArc;
+import opennlp.tools.parse_thicket.kernel_interface.TreeKernelRunner;
+import opennlp.tools.parse_thicket.matching.PT2ThicketPhraseBuilder;
+import opennlp.tools.parse_thicket.rhetoric_structure.RhetoricStructureArcsBuilder;
+
+import org.apache.commons.io.FileUtils;
+import org.jgrapht.Graph;
+import org.jgrapht.graph.DefaultEdge;
+import org.jgrapht.graph.SimpleGraph;
+
+
+
+
+
+
+
+
+
+
+import edu.stanford.nlp.trees.Tree;
+
+public class PT2ThicketPhraseBuilderExtrnlRST extends PT2ThicketPhraseBuilder{
+
+	private RhetoricStructureArcsBuilder rstBuilder = new RhetoricStructureArcsBuilder();
+	private ExternalRSTImporter externalRstBuilder = new ExternalRSTImporter();
+	private static Logger log = Logger
+		      .getLogger("opennlp.tools.parse_thicket.external_rst.PT2ThicketPhraseBuilderExtrnlRST");
+
+
+	/*
+	 * Building phrases takes a Parse Thicket and forms phrases for each sentence individually
+	 * Then based on built phrases and obtained arcs, it builds arcs for RST
+	 * Finally, based on all formed arcs, it extends phrases with thicket phrases
+	 */
+
+	public List<List<ParseTreeNode>> buildPT2ptPhrases(ParseThicket pt, String text, String externRSTpath) {
+		List<List<ParseTreeNode>> phrasesAllSent = new ArrayList<List<ParseTreeNode>> ();
+		Map<Integer, List<List<ParseTreeNode>>> sentNumPhrases = new HashMap<Integer, List<List<ParseTreeNode>>>();
+		// build regular phrases
+		for(int nSent=0; nSent<pt.getSentences().size(); nSent++){
+			List<ParseTreeNode> sentence = pt.getNodesThicket().get(nSent);
+			Tree ptree = pt.getSentences().get(nSent);
+			//ptree.pennPrint();
+			List<List<ParseTreeNode>> phrases = buildPT2ptPhrasesForASentence(ptree, sentence);
+			System.out.println(phrases);
+			phrasesAllSent.addAll(phrases);
+			sentNumPhrases.put(nSent, phrases);
+
+		}
+		String filename = "default.txt";
+		
+		try {
+			filename = text.split("/n")[0]+".txt";
+			FileUtils.writeStringToFile(new File(externRSTpath+"/"+filename), text, "utf-8");
+		} catch (Exception e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		}
+		// discover and add RST arcs
+		List<WordWordInterSentenceRelationArc> arcsRST =
+				rstBuilder.buildRSTArcsFromMarkersAndCorefs(pt.getArcs(), sentNumPhrases, pt);
+		String[] commandLine1 = new String[]{"python", "Discourse_Segmenter.py",  filename}, commandLine2=
+				new String[]{"python", "Discourse_Parser.py", "tmp.edu"};
+		new TreeKernelRunner().runEXE(commandLine1, externRSTpath);
+		new TreeKernelRunner().runEXE(commandLine2, externRSTpath);
+		
+		// TODO: code to run joty suite
+		List<RstNode> rstNodes = new ExternalRSTImporter().buildArrayOfRSTnodes(null, externRSTpath+"/tmp_doc.dis");
+	
+		// discover and add RST arcs
+		List<WordWordInterSentenceRelationArc> arcsRSTexternal = externalRstBuilder.buildRSTArcsFromRSTparser(  rstNodes, null, sentNumPhrases, pt );
+		System.out.println(arcsRST);
+
+
+		List<WordWordInterSentenceRelationArc> arcs = pt.getArcs();
+		arcs.addAll(arcsRST);
+		arcs.addAll(arcsRSTexternal);
+		pt.setArcs(arcs);
+
+
+		List<List<ParseTreeNode>> expandedPhrases = expandTowardsThicketPhrases(phrasesAllSent, pt.getArcs(), sentNumPhrases, pt);
+		return expandedPhrases;
+	}
+
+	
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/external_rst/ParseCorefBuilderWithNERandRST.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/external_rst/ParseCorefBuilderWithNERandRST.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/external_rst/ParseCorefBuilderWithNERandRST.java
new file mode 100644
index 0000000..9fe9524
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/external_rst/ParseCorefBuilderWithNERandRST.java
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.parse_thicket.external_rst;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.logging.Logger;
+
+import edu.arizona.sista.discourse.rstparser.DiscourseTree;
+import edu.arizona.sista.processors.CorefMention;
+import edu.arizona.sista.processors.Document;
+import edu.arizona.sista.processors.Processor;
+import edu.arizona.sista.processors.Sentence;
+import edu.arizona.sista.processors.corenlp.CoreNLPProcessor;
+import edu.arizona.sista.struct.DirectedGraphEdgeIterator;
+import edu.stanford.nlp.ie.AbstractSequenceClassifier;
+import edu.stanford.nlp.ie.crf.CRFClassifier;
+import edu.stanford.nlp.ling.CoreLabel;
+import edu.stanford.nlp.trees.Tree;
+import opennlp.tools.parse_thicket.ArcType;
+import opennlp.tools.parse_thicket.Pair;
+import opennlp.tools.parse_thicket.ParseThicket;
+import opennlp.tools.parse_thicket.ParseTreeNode;
+import opennlp.tools.parse_thicket.WordWordInterSentenceRelationArc;
+import opennlp.tools.parse_thicket.communicative_actions.CommunicativeActionsArcBuilder;
+import scala.Option;
+
+public class ParseCorefBuilderWithNERandRST {	
+	public Processor proc = null;
+	CommunicativeActionsArcBuilder caFinder = new CommunicativeActionsArcBuilder();
+	private static Logger log = Logger
+		      .getLogger("opennlp.tools.parse_thicket.external_rst.ParseCorefBuilderWithNERandRST");
+
+
+	AbstractSequenceClassifier<CoreLabel> classifier = null;
+
+	ParseCorefBuilderWithNERandRST() {
+		super();
+		classifier = CRFClassifier.getDefaultClassifier();
+		proc = new CoreNLPProcessor(true, true, 100);
+	}
+
+	public ParseThicketWithDiscourseTree buildParseThicket(String text){
+		List<Tree> ptTrees = new ArrayList<Tree>();
+		List<WordWordInterSentenceRelationArc> arcs = new ArrayList<WordWordInterSentenceRelationArc>();
+		List<List<ParseTreeNode>> nodesThicket = new ArrayList<List<ParseTreeNode>>();
+
+		Document doc=null;
+        try {
+	        doc = proc.annotate(text, false);
+        } catch (IllegalArgumentException iae) {
+        	log.severe("failed to parse text: "+text);
+        } catch (Exception e) {
+	        e.printStackTrace();
+        }
+        // failed to parse - skip this text
+		if (doc==null)
+			return null;
+		//java.lang.IllegalArgumentException
+		for (Sentence sentence: doc.sentences()) {
+			List<ParseTreeNode> sentenceNodes = new ArrayList<ParseTreeNode>();
+			String[] tokens= sentence.words();
+			for(int i=0; i< tokens.length; i++){
+				//sentence.startOffsets(), " "));
+				//sentence.endOffsets(), " "));
+				ParseTreeNode p = new ParseTreeNode(sentence.words()[i], sentence.tags().get()[i]);
+				p.setId(i+1);
+				if(sentence.entities().isDefined()){
+					p.setNe(sentence.entities().get()[i]);
+				}
+				if(sentence.norms().isDefined()){
+					//p.setNormalizedWord(sentence.norms().get()[i]);
+					p.setNormalizedWord(sentence.lemmas().get()[i]);
+				}
+				sentenceNodes.add(p);
+			}
+
+			if(sentence.dependencies().isDefined()) {
+				int i=0;
+				DirectedGraphEdgeIterator<String> iterator = new
+						DirectedGraphEdgeIterator<String>(sentence.dependencies().get());
+				while(iterator.hasNext()) {
+					scala.Tuple3<Object, Object, String> dep = iterator.next();
+					//System.out.println(" head:" + dep._1() + " modifier:" + dep._2() + " label:" + dep._3());
+					if (i>sentenceNodes.size()-1)
+						break;
+					ParseTreeNode p = sentenceNodes.get(i);
+					p.setHead(dep._1().toString());
+					p.setModifier(dep._2().toString());
+					p.setLabel(dep._3());
+					sentenceNodes.set(i, p);
+					i++;
+				}
+			}
+			if(sentence.syntacticTree().isDefined()) {
+				Tree tree = Tree.valueOf(sentence.syntacticTree().get().toString());
+				ptTrees.add(tree);
+				//tree.pennPrint();
+			}
+			nodesThicket.add(sentenceNodes);
+		}
+
+		if(doc.coreferenceChains().isDefined()) {
+			// these are scala.collection Iterator and Iterable (not Java!)
+			scala.collection.Iterator<scala.collection.Iterable<CorefMention>> chains = doc.coreferenceChains().get().getChains().iterator();
+			while(chains.hasNext()) {
+				scala.collection.Iterator<CorefMention> chain = chains.next().iterator();
+				//System.out.println("Found one coreference chain containing the following mentions:");
+				int numInChain = 0;
+				int[] niSentence = new int[4], niWord = new int[4], startOffset = new int[4], endOffset = new int[4];
+
+				while(chain.hasNext()) {
+					CorefMention mention = chain.next();
+					// note that all these offsets start at 0 too
+					niSentence[numInChain ] = mention.sentenceIndex();
+					niWord[numInChain ] = mention.headIndex();
+					startOffset[numInChain ] = mention.startOffset();
+					endOffset[numInChain ] = mention.endOffset();
+					if (numInChain>=4-1)
+						break;
+					numInChain++;
+					//" headIndex:" + mention.headIndex() +
+					//" startTokenOffset:" + mention.startOffset() +
+					//" endTokenOffset:" + mention.endOffset());
+				}
+				if (numInChain>0) { // more than a single mention
+					for(int i=0; i<numInChain; i++){
+						ArcType arcType = new ArcType("coref-", "", 0, 0);
+
+						WordWordInterSentenceRelationArc arc = 
+								new WordWordInterSentenceRelationArc(new Pair<Integer, Integer>(niSentence[i],niWord[i]), 
+										new Pair<Integer, Integer>(niSentence[i+1],niWord[i+1]), 
+									    startOffset[i]+"", startOffset[i+1]+"",
+	      	    					  arcType);
+						arcs.add(arc);
+					}
+				}
+			}
+		}
+
+
+		List<WordWordInterSentenceRelationArc> arcsCA = buildCAarcs(nodesThicket);
+		arcs.addAll(arcsCA);
+		ParseThicketWithDiscourseTree result = new ParseThicketWithDiscourseTree(ptTrees, arcs);
+
+		if(doc.discourseTree().isDefined()) {
+			Option<DiscourseTree> discourseTree = doc.discourseTree();
+
+			//scala.collection.immutable.List<DiscourseTree> scList = discourseTree.toList();
+			scala.collection.Iterator<DiscourseTree> iterator = discourseTree.iterator();
+			while(iterator.hasNext()) {
+				DiscourseTree dt = iterator.next();
+				result.setDt(dt);
+				List<WordWordInterSentenceRelationArc> rstArcs = new ArrayList<WordWordInterSentenceRelationArc>();
+				navigateDiscourseTree(dt, rstArcs, nodesThicket );
+				arcs.addAll(rstArcs);
+				System.out.println(dt);
+				System.out.println("first EDU = "+dt.firstEDU() + "| dt.firstSentence() = "+ dt.firstSentence() + 
+						" \n| last EDU = "+dt.lastEDU() + "| dt.lastSentence() = "+ dt.lastSentence() + 
+						" \n| dt.tokenCount() = " + dt.tokenCount() + "| dt.firstToken " + dt.firstToken() + 
+						" | dt.lastToken() "+ dt.lastToken() + "\n kind =" + dt.kind() + " | text = "+ dt.rawText());
+				StringBuilder sb = new StringBuilder(10000);
+				System.out.println(sb);
+			}
+		}
+
+		result.setOrigText(text);
+		result.setNodesThicket(nodesThicket);
+		
+		result.setDtDump(); // sets the DT representation for TK learning
+		return result;
+	}
+
+	public List<WordWordInterSentenceRelationArc> buildCAarcs(
+			List<List<ParseTreeNode>> nodesThicket) {
+		List<WordWordInterSentenceRelationArc> arcs = new ArrayList<WordWordInterSentenceRelationArc>();
+
+		for(int sentI=0; sentI<nodesThicket.size(); sentI++){
+			for(int sentJ=sentI+1; sentJ<nodesThicket.size(); sentJ++){
+				List<ParseTreeNode> sentenceI = nodesThicket.get(sentI), 
+						sentenceJ = nodesThicket.get(sentJ);
+				Pair<String, Integer[]> caI = caFinder.findCAInSentence(sentenceI);
+				Pair<String, Integer[]> caJ = caFinder.findCAInSentence(sentenceJ);
+				int indexCA1 = caFinder.findCAIndexInSentence(sentenceI);
+				int indexCA2 = caFinder.findCAIndexInSentence(sentenceJ);
+				if (caI==null || caJ==null)
+					continue;
+				Pair<String, Integer[]> caGen = caFinder.generalize(caI, caJ).get(0);
+
+				ArcType arcType = new ArcType("ca", 
+						caGen.getFirst().toString()+printNumArray(caGen.getSecond()), 0, 0);
+				WordWordInterSentenceRelationArc arc = 
+						new WordWordInterSentenceRelationArc(new Pair<Integer, Integer>(sentI,indexCA1), 
+								new Pair<Integer, Integer>(sentJ,indexCA2), caI.getFirst(), caJ.getFirst(), 
+								arcType);
+				arcs.add(arc);
+
+			}
+		}
+
+		return arcs;
+	}
+
+	private String printNumArray(Integer[] arr){
+		StringBuffer buf = new StringBuffer();
+		for(Integer i: arr){
+			buf.append(Integer.toString(i)+ " ");
+		}
+		return buf.toString();
+	}
+
+	// creates a list of Arcs objects 'arcs' from the descourse tree dt, using the list of sentences 'nodesThicket' to identify words 
+	// for nodes being connected with these arcs
+	private void navigateDiscourseTree(DiscourseTree dt, List<WordWordInterSentenceRelationArc> arcs,  List<List<ParseTreeNode>> nodesThicket  ) {
+		if (dt.isTerminal()) {
+			return;
+		} else {
+			ArcType arcType = new ArcType("rst", 
+					dt.relationLabel()+ "=>" + dt.kind(), Boolean.compare(dt.relationDirection().equals("LeftToRight"), true),0);
+			String lemmaFrom = nodesThicket.get(dt.firstSentence()).get(dt.firstToken().copy$default$2()).getWord();
+			String lemmaTo = nodesThicket.get(dt.lastSentence()).get(dt.lastToken().copy$default$2()-1).getWord();
+			
+			WordWordInterSentenceRelationArc arc = 
+					new WordWordInterSentenceRelationArc(new Pair<Integer, Integer>(dt.firstToken().copy$default$1(), dt.firstToken().copy$default$2()), 
+							new Pair<Integer, Integer>(dt.lastToken().copy$default$1(), dt.lastToken().copy$default$2()), 
+							lemmaFrom,lemmaTo, 
+							arcType);
+			System.out.println(arc);
+			arcs.add(arc);
+			DiscourseTree[] kids = dt.children();
+			if (kids != null) {
+				for (DiscourseTree kid : kids) {
+					navigateDiscourseTree(kid, arcs, nodesThicket);
+				}
+			}
+			return ;
+		}
+	}
+
+	public static void main(String[] args){
+		ParseCorefBuilderWithNERandRST builder = new ParseCorefBuilderWithNERandRST ();
+		String text = "I thought I d tell you a little about what I like to write. And I like to immerse myself in my topics. I just like to dive right in and become sort of a human guinea pig. And I see my life as a series of experiments. So , I work for Esquire magazine , and a couple of years ago I wrote an article called  My Outsourced Life ,  where I hired a team of people in Bangalore , India , to live my life for me. "
+		+ "So they answered my emails. They answered my phone. ";
+		
+		ParseThicket pt = builder.buildParseThicket(text);
+		pt = builder.buildParseThicket(
+				"Dutch accident investigators say that evidence points to pro-Russian rebels as being responsible for shooting down plane. The report indicates where the missile was fired from and identifies who was in control of the territory and pins the downing of the plane on the pro-Russian rebels. "+
+						"However, the Investigative Committee of the Russian Federation believes that the plane was hit by a missile from the air which was not produced in Russia. "+
+						"At the same time, rebels deny that they controlled the territory from which the missile was supposedly fired."
+				);
+	}
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/external_rst/ParseThicketWithDiscourseTree.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/external_rst/ParseThicketWithDiscourseTree.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/external_rst/ParseThicketWithDiscourseTree.java
new file mode 100644
index 0000000..44c843c
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/external_rst/ParseThicketWithDiscourseTree.java
@@ -0,0 +1,284 @@
+package opennlp.tools.parse_thicket.external_rst;
+
+import java.util.List;
+
+import edu.arizona.sista.discourse.rstparser.DiscourseTree;
+import edu.stanford.nlp.trees.Tree;
+import opennlp.tools.parse_thicket.Pair;
+import opennlp.tools.parse_thicket.ParseThicket;
+import opennlp.tools.parse_thicket.ParseTreeNode;
+import opennlp.tools.parse_thicket.VerbNetProcessor;
+import opennlp.tools.parse_thicket.WordWordInterSentenceRelationArc;
+import opennlp.tools.parse_thicket.kernel_interface.TreeExtenderByAnotherLinkedTree;
+
+/*
+ * This is subclass of ParseThicket with the focus on Discourse Tree
+ * It produces a representation of discourse tree for tree kernel learning
+ */
+
+public class ParseThicketWithDiscourseTree extends ParseThicket {
+	private DiscourseTree dt;
+	private String dtDump;
+	private String dtDumpWithPOS;
+	private String dtDumpWithEmbeddedTrees;
+	private String dtDumpWithVerbNet;
+
+	private TreeExtenderByAnotherLinkedTree extender = new TreeExtenderByAnotherLinkedTree();
+	private VerbNetProcessor verbBuilder = VerbNetProcessor.getInstance(null);
+
+	public DiscourseTree getDt() {
+		return dt;
+	}
+	// sets the highest level DT (under further iterations does not set anything)
+	public void setDt(DiscourseTree dt) {
+		if (this.dt==null)
+			this.dt = dt;
+	}
+
+	public ParseThicketWithDiscourseTree(List<Tree> ptTrees, List<WordWordInterSentenceRelationArc> barcs) {
+		super(ptTrees, barcs);
+	}
+
+	public void setDtDump(){
+		StringBuilder sb = new StringBuilder(100000);
+		StringBuilder res = toStringBuilderDTWithPOSSeq(sb, this.dt);
+		dtDumpWithPOS = res.toString();
+
+		sb = new StringBuilder(100000);
+		res = toStringBuilderDT(sb, this.dt);
+		dtDump = res.toString();
+
+		sb = new StringBuilder(100000);
+		res = toStringBuilderDTWithEmbeddedTrees(sb, this.dt);
+		dtDumpWithEmbeddedTrees = res.toString();
+
+		sb = new StringBuilder(100000);
+		res = toStringBuilderDTWithVerbNet(sb, this.dt);
+		dtDumpWithVerbNet = res.toString();
+	}
+	// basic representation of discourse tree 
+	private StringBuilder toStringBuilderDT(StringBuilder sb, DiscourseTree dt) {
+		if (dt.isTerminal()) {
+			if (dt.relationLabel() != null) {
+				sb.append(dt.relationLabel());
+				//sb.append("("+dt.rawText()+")");
+				scala.collection.mutable.StringBuilder sbs = new scala.collection.mutable.StringBuilder(100);
+
+				dt.print(sbs, 0, false, true);
+				String text  =  sbs.replaceAllLiterally("Nucleus TEXT:", "(");
+				text = text.substring(0, text.length()-1)+")";
+				sb.append(text);
+			}
+			return sb;
+		} else {
+			sb.append('(');
+			if (dt.relationLabel() != null) {
+				sb.append(dt.relationLabel());
+			}
+			DiscourseTree[] kids = dt.children();
+			if (kids != null) {
+				for (DiscourseTree kid : kids) {
+					sb.append(' ');
+					toStringBuilderDT(sb, kid);
+				}
+			}
+			return sb.append(')');
+		}
+	}
+
+	private StringBuilder toStringBuilderDTWithPOSSeq(StringBuilder sb, DiscourseTree dt) {
+		if (dt.isTerminal()) {
+			if (dt.relationLabel() != null && dt.relationLabel().length()>2) {
+				sb.append(dt.relationLabel());
+				// different StrBuilder for trees from scala
+				scala.collection.mutable.StringBuilder sbs = new scala.collection.mutable.StringBuilder(100);
+				dt.print(sbs, 0, false, true);
+				String text  =  sbs.replaceAllLiterally("Nucleus TEXT:", "");
+				//text = text.substring(0, text.length()-1)+"";
+				String textDump = substituteTextWithPOStext(text, this.getNodesThicket().get(dt.firstToken().copy$default$1()));
+				sb.append(textDump);
+			}
+			return sb;
+		} else {
+			sb.append('(');
+			if (dt.relationLabel() != null) {
+				sb.append(dt.relationLabel());
+			}
+			DiscourseTree[] kids = dt.children();
+			if (kids != null) {
+				for (DiscourseTree kid : kids) {
+					sb.append(' ');
+					toStringBuilderDTWithPOSSeq(sb, kid);
+				}
+			}
+			return sb.append(')');
+		}
+	}
+
+	private String substituteTextWithPOStext(String text, List<ParseTreeNode> list) {
+		boolean bMatch = false;
+		String[] tokens = text.split(" ");
+		for(int offset = 0; offset<list.size(); offset++ ){	    	
+			List<ParseTreeNode> subList = list.subList(offset, tokens.length+offset);
+			int count = 0;
+			bMatch = true; // if at least one mismatch
+			for(ParseTreeNode n: subList){
+				if (!n.getWord().equals(tokens[count])){
+					bMatch = false;
+					break;
+				} else 
+					count++;
+				if (count>3)
+					break;
+			}
+			if (bMatch){
+				return //"(" + 
+						ParseTreeNode.toTreeRepresentationString(subList); // + ")";
+			}
+		}
+		return null;
+	}
+
+	private StringBuilder toStringBuilderDTWithEmbeddedTrees(StringBuilder sb, DiscourseTree dt) {
+		if (dt.isTerminal()) {
+			if (dt.relationLabel() != null && dt.relationLabel().length()>2) {
+				sb.append(dt.relationLabel());
+				//sb.append("("+dt.rawText()+")");
+				scala.collection.mutable.StringBuilder sbs = new scala.collection.mutable.StringBuilder(100);
+
+				dt.print(sbs, 0, false, true);
+				String text  =  sbs.replaceAllLiterally("Nucleus TEXT:", "");
+				//text = text.substring(0, text.length()-1)+"";
+				substituteTextWithParseTree(sb, text, this.getSentenceTrees().get(dt.firstToken().copy$default$1()));
+			}
+			return sb;
+		} else {
+			sb.append('(');
+			if (dt.relationLabel() != null) {
+				sb.append(dt.relationLabel());
+			}
+			DiscourseTree[] kids = dt.children();
+			if (kids != null) {
+				for (DiscourseTree kid : kids) {
+					sb.append(' ');
+					toStringBuilderDTWithEmbeddedTrees(sb, kid);
+				}
+			}
+			return sb.append(')');
+		}
+	}
+	private void substituteTextWithParseTree(StringBuilder sb, String text, Tree sentenceTree) {
+		String[] tokens = text.split(" ");
+		List<Tree> foundTrees = null;
+		if (tokens.length>1){
+			foundTrees = 
+					extender.getASubtreeWithRootAsNodeForWord1(sentenceTree, sentenceTree, new String[]{tokens[0], tokens[1]});
+		}
+		else{
+			foundTrees = 
+					extender.getASubtreeWithRootAsNodeForWord1(sentenceTree, sentenceTree, new String[]{tokens[0]});
+
+		}
+
+		if (foundTrees == null || foundTrees.size()<1)
+			return;
+
+		extender.toStringBuilder(sb, foundTrees.get(0));
+
+	}
+
+	private StringBuilder toStringBuilderDTWithVerbNet(StringBuilder sb, DiscourseTree dt) {
+		if (dt.isTerminal()) {
+			if (dt.relationLabel() != null && dt.relationLabel().length()>2) {
+				sb.append(dt.relationLabel());
+				//sb.append("("+dt.rawText()+")");
+				scala.collection.mutable.StringBuilder sbs = new scala.collection.mutable.StringBuilder(100);
+
+				dt.print(sbs, 0, false, true);
+				String text  =  sbs.replaceAllLiterally("Nucleus TEXT:", "");
+				String textDump = null;
+				if (text.split(" ").length<100) // if not TOO long, more informative substitution, including VerbNets
+					textDump = substituteTextWithPOStextVerbNet(text, this.getNodesThicket().get(dt.firstToken().copy$default$1()));
+				else // otherwise just lemma-POS chains
+					textDump = substituteTextWithPOStext(text, this.getNodesThicket().get(dt.firstToken().copy$default$1()));
+				
+					
+				sb.append(textDump);
+			}
+			return sb;
+		} else {
+			sb.append('(');
+			if (dt.relationLabel() != null) {
+				sb.append(dt.relationLabel());
+			}
+			DiscourseTree[] kids = dt.children();
+			if (kids != null) {
+				for (DiscourseTree kid : kids) {
+					sb.append(' ');
+					toStringBuilderDTWithVerbNet(sb, kid);
+				}
+			}
+			return sb.append(')');
+		}
+	}
+
+	// substitutes lemma-POS pair instead of just lemma
+	// in case of verb provides moe detailed info
+	private String substituteTextWithPOStextVerbNet(String text, List<ParseTreeNode> list) {
+		boolean bMatch = false;
+		String[] tokens = text.split(" ");
+		for(int offset = 0; offset<list.size(); offset++ ){	    	
+			List<ParseTreeNode> subList = list.subList(offset, tokens.length+offset);
+			int count = 0;
+			bMatch = true; // if at least one mismatch
+			for(ParseTreeNode n: subList){
+				if (!n.getWord().equals(tokens[count])){
+					bMatch = false;
+					break;
+				} else 
+					count++;
+				if (count>3) // three tokens is enough for alignment
+					break;
+			}
+			// alignment found; now 
+			if (bMatch){
+				StringBuilder buf = new StringBuilder();
+				for(ParseTreeNode ch: subList){
+					try {
+	                    if (ch.getPos().startsWith(".") || ch.getPos().startsWith(",") || ch.getPos().startsWith(";") || ch.getPos().startsWith("!"))
+	                    	continue;
+	                    if (ch.getPos().startsWith("VB") && ch.getNormalizedWord()!=null){ // do more info for verbs
+	                    	StringBuilder verbRepr = verbBuilder.
+	                    			buildTreeRepresentationForTreeKernelLearning(ch.getNormalizedWord());
+	                    	if (verbRepr!=null)
+	                    		buf.append(" ("+verbRepr+") ");
+	                    	else
+	                    		buf.append( "("+ch.getWord()+ " " + ch.getPos() + ")" );
+	                    } else { // other than verb
+	                    	buf.append( "("+ch.getWord()+ " " + ch.getPos() + ")" );
+	                    }
+                    } catch (Exception e) {
+	                    e.printStackTrace();
+                    }
+				}
+				return buf.toString().trim();
+			}
+		}
+		return null;
+	}
+
+	public String getDtDump() {
+		return this.dtDump;
+	}
+	public String getDtDumpWithPOS() {
+		return this.dtDumpWithPOS;
+	}
+
+	public String getDtDumpWithEmbeddedTrees() {
+		return this.dtDumpWithEmbeddedTrees;
+	}
+	
+	public String getDtDumpWithVerbNet() {
+		return this.dtDumpWithVerbNet;
+	}
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/external_rst/RstNode.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/external_rst/RstNode.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/external_rst/RstNode.java
new file mode 100644
index 0000000..61e8f13
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/external_rst/RstNode.java
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.parse_thicket.external_rst;
+
+import org.apache.commons.lang.StringUtils;
+
+import opennlp.tools.similarity.apps.utils.Pair;
+
+public class RstNode {
+	Boolean isNucleus;
+	Pair<Integer, Integer> span;
+	Integer leaf;
+	String rel2par;
+	String text;
+	Integer level;
+	
+	public Boolean getIsNucleus() {
+		return isNucleus;
+	}
+	public void setIsNucleus(Boolean isNucleus) {
+		this.isNucleus = isNucleus;
+	}
+	public Pair<Integer, Integer> getSpan() {
+		return span;
+	}
+	public void setSpan(Pair<Integer, Integer> span) {
+		this.span = span;
+	}
+	public Integer getLeaf() {
+		return leaf;
+	}
+	public void setLeaf(Integer leaf) {
+		this.leaf = leaf;
+	}
+	public String getRel2par() {
+		return rel2par;
+	}
+	public void setRel2par(String rel2par) {
+		this.rel2par = rel2par;
+	}
+	public String getText() {
+		return text;
+	}
+	public void setText(String text) {
+		this.text = text;
+	}
+	
+	public String toString() {
+		String ret = "";
+		if (isNucleus!=null && isNucleus)
+			ret+="Nucleus ";
+		if (span!=null)
+			ret+="["+span.getFirst()+" "+ span.getSecond()+"]";
+		ret += " >> "+ rel2par;
+		if (text!=null)
+			ret+= " >> "+text;
+		return ret;
+	}
+	public RstNode(String line) {
+		if (StringUtils.trim(line).startsWith(")"))
+			return;
+		
+
+		level = line.indexOf("(");
+		line = line.substring(line.indexOf("(")+2);
+		
+		isNucleus = line.substring(0, line.indexOf("(")).indexOf("Nucleus")>-1;
+		line = line.substring(line.indexOf("(")+1);
+		if (line.startsWith("span")){
+			line = line.substring(5);
+			try {
+				span = new Pair<Integer, Integer>();
+				String[] spanStr = line.substring(0, line.indexOf(")")).split(" "); 
+				span.setFirst(Integer.parseInt(spanStr[0]));
+				span.setSecond(Integer.parseInt(spanStr[1]));
+			} catch (Exception e) {
+				e.printStackTrace();
+			}
+			
+		} else if (line.startsWith("leaf")){
+			try {
+				String leafStr = line.substring(5, line.indexOf(")"));
+				leaf = Integer.parseInt(leafStr);
+			} catch (Exception e) {
+				e.printStackTrace();
+			}
+			
+		} else System.err.println("Problem parsing RST results: '"+line);
+		
+		line = line.substring(line.indexOf("rel2par")+8);
+		rel2par = line.substring(0, line.indexOf(")")).trim();
+		
+		text = StringUtils.substringBetween(line, "_!", "_!)");
+
+		
+	}
+
+	 public static void main(String[] args){
+		 RstNode n1 = new RstNode("        ( Nucleus (leaf 7) (rel2par span) (text _!that it usually takes a day_!) )"),
+		 n2 = new RstNode("       )"),
+		 n3 = new RstNode("          ( Satellite (span 15 16) (rel2par Explanation)");
+		 
+	 }
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/BracesProcessor.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/BracesProcessor.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/BracesProcessor.java
new file mode 100644
index 0000000..b41cd46
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/BracesProcessor.java
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.parse_thicket.kernel_interface;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Stack;
+
+//import org.apache.commons.io.FileUtils;
+
+public class BracesProcessor {
+	private static final int MIN_BRACES_CNT = 5;
+
+	private static final char L_PAREN    = '(';
+	private static final char R_PAREN    = ')';
+	private static final char L_BRACE    = '{';
+	private static final char R_BRACE    = '}';
+	private static final char L_BRACKET  = '[';
+	private static final char R_BRACKET  = ']';
+	private  Stack<Character> stackIncremental = new Stack<Character>();
+	private int count = 0;
+	private Boolean balancedSoFar = true;
+
+	public Boolean getBalancedBracesResult(){
+		if (balancedSoFar)
+			return (stackIncremental.isEmpty() && count> MIN_BRACES_CNT);
+		else 
+			return false;
+	}
+
+	public void analyzeBalancedBracesAddPortionIncremental(String s) {
+
+
+		for (int i = 0; i < s.length(); i++) {
+
+			if      (s.charAt(i) == L_PAREN)   {
+				stackIncremental.push(L_PAREN);
+				count++;
+
+			}
+
+			else if (s.charAt(i) == L_BRACE)   {
+				stackIncremental.push(L_BRACE);
+				count++;
+			}
+
+			else if (s.charAt(i) == L_BRACKET){
+				stackIncremental.push(L_BRACKET);
+				count++;
+			}
+
+			else if (s.charAt(i) == R_PAREN) {
+				if (stackIncremental.isEmpty())        balancedSoFar = false;
+				if (stackIncremental.pop() != L_PAREN) balancedSoFar = false;
+			}
+
+			else if (s.charAt(i) == R_BRACE) {
+				if (stackIncremental.isEmpty())        balancedSoFar = false;
+				if (stackIncremental.pop() != L_BRACE) balancedSoFar = false;
+			}
+
+			else if (s.charAt(i) == R_BRACKET) {
+				if (stackIncremental.isEmpty())        balancedSoFar = false;
+				if (stackIncremental.pop() != L_BRACKET) balancedSoFar = false;
+			}
+
+			// ignore all other characters
+
+		}
+
+	}
+
+	public static boolean isBalanced(String s) {
+		int count = 0;
+		Stack<Character> stack = new Stack<Character>();
+		for (int i = 0; i < s.length(); i++) {
+
+			if      (s.charAt(i) == L_PAREN)   {
+				stack.push(L_PAREN);
+				count++;
+
+			}
+
+			else if (s.charAt(i) == L_BRACE)   {
+				stack.push(L_BRACE);
+				count++;
+			}
+
+			else if (s.charAt(i) == L_BRACKET){
+				stack.push(L_BRACKET);
+				count++;
+			}
+
+			else if (s.charAt(i) == R_PAREN) {
+				if (stack.isEmpty())        return false;
+				if (stack.pop() != L_PAREN) return false;
+			}
+
+			else if (s.charAt(i) == R_BRACE) {
+				if (stack.isEmpty())        return false;
+				if (stack.pop() != L_BRACE) return false;
+			}
+
+			else if (s.charAt(i) == R_BRACKET) {
+				if (stack.isEmpty())        return false;
+				if (stack.pop() != L_BRACKET) return false;
+			}
+
+			// ignore all other characters
+
+		}
+		return (stack.isEmpty());
+	}
+
+	public static boolean checkParentesis(String str)
+	{
+		if (str.isEmpty())
+			return true;
+
+		Stack<Character> stack = new Stack<Character>();
+		for (int i = 0; i < str.length(); i++)
+		{
+			char current = str.charAt(i);
+			if (current == '{' || current == '(' || current == '[')
+			{
+				stack.push(current);
+			}
+
+
+			if (current == '}' || current == ')' || current == ']')
+			{
+				if (stack.isEmpty())
+					return false;
+
+				char last = stack.peek();
+				if (current == '}' && (last == '{' || current == ')')
+						&& last == '(' || (current == ']'
+						&& last == '['))
+					stack.pop();
+				else 
+					return false;
+			}
+
+		}
+
+		return stack.isEmpty();
+	}
+
+	public static boolean isParenthesisMatch(String str) {
+		Stack<Character> stack = new Stack<Character>();
+
+		char c;
+		for(int i=0; i < str.length(); i++) {
+			c = str.charAt(i);
+
+			if(c == '{')
+				return false;
+
+			if(c == '(')
+				stack.push(c);
+
+			if(c == '{') {
+				stack.push(c);
+				if(c == '}')
+					if(stack.empty())
+						return false;
+					else if(stack.peek() == '{')
+						stack.pop();
+			}
+			else if(c == ')')
+				if(stack.empty())
+					return false;
+				else if(stack.peek() == '(')
+					stack.pop();
+				else
+					return false;
+		}
+		return stack.empty();
+	}
+
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/DescriptiveParagraphFromDocExtractor.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/DescriptiveParagraphFromDocExtractor.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/DescriptiveParagraphFromDocExtractor.java
new file mode 100644
index 0000000..1c07719
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/DescriptiveParagraphFromDocExtractor.java
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.parse_thicket.kernel_interface;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.tika.Tika;
+import org.apache.tika.exception.TikaException;
+
+public class DescriptiveParagraphFromDocExtractor {
+	protected static  Tika tika = new Tika();
+	private static int MIN_PARA_LENGTH = 200, //120, 
+			MIN_NUM_WORDS=15, 
+			MAX_PARA_LENGTH = 500, //200 
+			TEXT_PORTION_FOR_ANALYSIS = 20000, 
+			MAX_PARA_OUTPUT=20;
+	public static String getFirstParagraphFromFile(File f) {
+
+		String text = "";
+		try {
+			try {
+				text = tika.parseToString(f);
+			} catch (TikaException e) {
+				// TODO Auto-generated catch block
+				e.printStackTrace();
+			} 
+			//text = FileUtils.readFileToString(f, null);
+			if (text.length()>TEXT_PORTION_FOR_ANALYSIS)
+				text = text.substring(0, TEXT_PORTION_FOR_ANALYSIS);
+			float avgSentSizeThr = (float)MIN_PARA_LENGTH/4f; //2f
+			String[] portions = text.split("\\.\\n");
+			for(String p: portions){
+				float avgSentSize = (float)p.length()/(float)p.split("\\n\\n").length;
+
+				if (p.length()> MIN_PARA_LENGTH && p.split(" ").length>MIN_NUM_WORDS &&
+						avgSentSize > avgSentSizeThr &&  p.length() < MAX_PARA_LENGTH){
+					return normalizePara(p);
+				}
+			}
+		} catch (IOException e) {
+			e.printStackTrace();
+		}
+		// if not a suitable paragraph found, return the whole text
+		if (text.length()>150)
+			text = text.substring(0, 150);
+		return text;
+	}
+
+	public static List<String> getLongParagraphsFromFile(File f) {
+		List<String> results = new ArrayList<String>();
+		String text = "";
+		try {
+			try {
+				text = tika.parseToString(f);
+			} catch (TikaException e) {
+				// TODO Auto-generated catch block
+				e.printStackTrace();
+			} 
+			//text = FileUtils.readFileToString(f, null);
+			if (text.length()>TEXT_PORTION_FOR_ANALYSIS)
+				text = text.substring(0, TEXT_PORTION_FOR_ANALYSIS);
+			float avgSentSizeThr = (float)MIN_PARA_LENGTH/4f; //2f
+			String[] portions = text.split("\\.\\n");
+			if (portions.length<2)
+				portions = text.split("\\n\\n");
+			if (portions.length<2)
+				portions = text.split("\\n \\n");
+			if (portions.length<2){
+				String[] sentences = text.replace('.','&').split(" & ");
+				List<String> portionsLst = new ArrayList<String>();
+				int totalChars = 0;
+				String buffer = "";
+				for(String sent: sentences){
+					totalChars+=sent.length();
+					if (totalChars>MAX_PARA_LENGTH){
+						portionsLst.add(buffer);
+						buffer="";
+						totalChars = 0;
+					} else {
+						buffer+= sent + ". ";
+					}
+				}
+				portions = portionsLst.toArray(new String[0]);
+			}
+			for(String p: portions){
+				try {
+					float avgSentSize = (float)p.length()/(float)p.split("\\n\\n").length;
+
+					if (p.length()> MIN_PARA_LENGTH && p.split(" ").length>MIN_NUM_WORDS &&
+							avgSentSize > avgSentSizeThr) {  
+						if (p.length() < MAX_PARA_LENGTH){
+							results.add(normalizePara(p)); 
+						}
+						else { // reduce length to the latest '.' in substring
+							
+							String pReduced = p;
+							if (p.length()>= MAX_PARA_LENGTH+80)
+								pReduced = p.substring(0, MAX_PARA_LENGTH+80);
+							int indexPeriod = pReduced.lastIndexOf('.');
+							if (indexPeriod>-1){
+								pReduced = pReduced.substring(0, indexPeriod);
+							}
+							results.add(normalizePara(pReduced));
+						}
+						if (results.size()>MAX_PARA_OUTPUT)
+							break;
+					}
+				} catch (Exception e) {
+					e.printStackTrace();
+				}
+			}
+			if (results.size()<1){
+				if (text.length()>= MAX_PARA_LENGTH+80)
+					text = text.substring(0, MAX_PARA_LENGTH+80);
+				results.add(text);
+			}
+
+		} catch (IOException e) {
+			e.printStackTrace();
+		}
+		if (results.size()<1){
+			System.err.println("Failed to extract text from "+f.getName());
+		}
+
+		return results;
+	}
+
+	private static String normalizePara(String p){
+		p = p.replaceAll("\\n", " ").replaceAll("\\.\\.", " ").replaceAll("  ", " ");
+		p = p.replaceAll("[^A-Za-z0-9 _\\.,\\!]", "");
+		return p;
+	}
+
+	public static void main(String args[]){
+		List<String> results = getLongParagraphsFromFile(new File(
+				"/Users/borisgalitsky/Documents/workspace/deepContentInspection/src/test/resources/1k/design_doc_posNeg/pos/2IP40 Detail Design Document.pdf"
+				//+ " Online Screening Tool - Delottie.pdf"
+				));
+		System.out.println(results);
+
+		String res = getFirstParagraphFromFile(new File("/Users/borisgalitsky/Documents/workspace/deepContentInspection/src/test/resources/1k/"
+				+ "design_doc/2004Schalk_BCI2000Implementation.pdf"));
+		System.out.println(res);
+		results = getLongParagraphsFromFile(new File("/Users/borisgalitsky/Documents/workspace/deepContentInspection/src/test/resources/1k/"
+				+ "design_doc/2004Schalk_BCI2000Implementation.pdf"));
+		System.out.println(results);
+
+	}
+}


[39/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/44FictAndersenH_Mermaid_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/44FictAndersenH_Mermaid_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/44FictAndersenH_Mermaid_EN.txt.txt
new file mode 100644
index 0000000..d93d5cc
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/44FictAndersenH_Mermaid_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff The little mermaid Far out in the ocean , where the water is as blue as the prettiest cornflower , and as clear as crystal , it is very , very deep ; so deep , indeed , that no cable could fathom it : many church steeples , piled one upon another , would not reach from the ground beneath to the surface of the water above . There dwell the Sea King and his subjects . We must not imagine that there is nothing at the bottom of the sea but bare yellow sand . No , indeed ; the most singular flowers and plants grow there ; the leaves and stems of which are so pliant , that the slightest agitation of the water causes them to stir as if they had life . Fishes , both large and small , glide between the branches , as birds fly among the trees here upon land . In the deepest spot of all , stands the castle of the Sea King . Its walls are built of coral , and the long , gothic windows are of the clearest amber . The roof is formed of shells , that open and close as the water flows over them
  . Their appearance is very beautiful , for in each lies a glittering pearl , which would be fit for the diadem of a queen . The Sea King had been a widower for many years , and his aged mother kept house for him . She was a very wise woman , and exceedingly proud of her high birth ; on that account she wore twelve oysters on her tail ; while others , also of high rank , were only allowed to wear six . She was , however , deserving of very great praise , especially for her care of the little sea-princesses , her grand-daughters . They were six beautiful children ; but the youngest was the prettiest of them all ; her skin was as clear and delicate as a rose-leaf , and her eyes as blue as the deepest sea ; but , like all the others , she had no feet , and her body ended in a fish 's tail . All day long they played in the great halls of the castle , or among the living flowers that grew out of the walls . The large amber windows were open , and the fish swam in , just as the swallows f
 ly into our houses when we open the windows , excepting that the fishes swam up to the princesses , ate out of their hands , and allowed themselves to be stroked . Outside the castle there was a beautiful garden , in which grew bright red and dark blue flowers , and blossoms like flames of fire ; the fruit glittered like gold , and the leaves and stems waved to and fro continually . The earth itself was the finest sand , but blue as the flame of burning sulphur . Over everything lay a peculiar blue radiance , as if it were surrounded by the air from above , through which the blue sky shone , instead of the dark depths of the sea . In calm weather the sun could be seen , looking like a purple flower , with the light streaming from the calyx . Each of the young princesses had a little plot of ground in the garden , where she might dig and plant as she pleased . One arranged her flower-bed into the form of a whale ; another thought it better to make hers like the figure of a little mer
 maid ; but that of the youngest was round like the sun , and contained flowers as red as his rays at sunset . She was a strange child , quiet and thoughtful ; and while her sisters would be delighted with the wonderful things which they obtained from the wrecks of vessels , she cared for nothing but her pretty red flowers , like the sun , excepting a beautiful marble statue . It was the representation of a handsome boy , carved out of pure white stone , which had fallen to the bottom of the sea from a wreck . She planted by the statue a rose-colored weeping willow . It grew splendidly , and very soon hung its fresh branches over the statue , almost down to the blue sands . The shadow had a violet tint , and waved to and fro like the branches ; it seemed as if the crown of the tree and the root were at play , and trying to kiss each other . Nothing gave her so much pleasure as to hear about the world above the sea . She made her old grandmother tell her all she knew of the ships and 
 of the towns , the people and the animals . To her it seemed most wonderful and beautiful to hear that the flowers of the land should have fragrance , and not those below the sea ; that the trees of the forest should be green ; and that the fishes among the trees could sing so sweetly , that it was quite a pleasure to hear them . Her grandmother called the little birds fishes , or she would not have understood her ; for she had never seen birds . \u201c When you have reached your fifteenth year , \u201d said the grand-mother , \u201c you will have permission to rise up out of the sea , to sit on the rocks in the moonlight , while the great ships are sailing by ; and then you will see both forests and towns . \u201d In the following year , one of the sisters would be fifteen : but as each was a year younger than the other , the youngest would have to wait five years before her turn came to rise up from the bottom of the ocean , and see the earth as we do . However , each promised to tell the oth
 ers what she saw on her first visit , and what she thought the most beautiful ; for their grandmother could not tell them enough ; there were so many things on which they wanted information . None of them longed so much for her turn to come as the youngest , she who had the longest time to wait , and who was so quiet and thoughtful . Many nights she stood by the open window , looking up through the dark blue water , and watching the fish as they splashed about with their fins and tails . She could see the moon and stars shining faintly ; but through the water they looked larger than they do to our eyes . When something like a black cloud passed between her and them , she knew that it was either a whale swimming over her head , or a ship full of human beings , who never imagined that a pretty little mermaid was standing beneath them , holding out her white hands towards the keel of their ship . As soon as the eldest was fifteen , she was allowed to rise to the surface of the ocean . 
 When she came back , she had hundreds of things to talk about ; but the most beautiful , she said , was to lie in the moonlight , on a sandbank , in the quiet sea , near the coast , and to gaze on a large town nearby , where the lights were twinkling like hundreds of stars ; to listen to the sounds of the music , the noise of carriages , and the voices of human beings , and then to hear the merry bells peal out from the church steeples ; and because she could not go near to all those wonderful things , she longed for them more than ever . Oh , did not the youngest sister listen eagerly to all these descriptions ? and afterwards , when she stood at the open window looking up through the dark blue water , she thought of the great city , with all its bustle and noise , and even fancied she could hear the sound of the church bells , down in the depths of the sea . In another year the second sister received permission to rise to the surface of the water , and to swim about where she plea
 sed . She rose just as the sun was setting , and this , she said , was the most beautiful sight of all . The whole sky looked like gold , while violet and rose-colored clouds , which she could not describe , floated over her ; and , still more rapidly than the clouds , flew a large flock of wild swans towards the setting sun , looking like a long white veil across the sea . She also swam towards the sun ; but it sunk into the waves , and the rosy tints faded from the clouds and from the sea . The third sister 's turn followed ; she was the boldest of them all , and she swam up a broad river that emptied itself into the sea . On the banks she saw green hills covered with beautiful vines ; palaces and castles peeped out from amid the proud trees of the forest ; she heard the birds singing , and the rays of the sun were so powerful that she was obliged often to dive down under the water to cool her burning face . In a narrow creek she found a whole troop of little human children , quit
 e naked , and sporting about in the water ; she wanted to play with them , but they fled in a great fright ; and then a little black animal came to the water ; it was a dog , but she did not know that , for she had never before seen one . This animal barked at her so terribly that she became frightened , and rushed back to the open sea . But she said she should never forget the beautiful forest , the green hills , and the pretty little children who could swim in the water , although they had not fish 's tails . The fourth sister was more timid ; she remained in the midst of the sea , but she said it was quite as beautiful there as nearer the land . She could see for so many miles around her , and the sky above looked like a bell of glass . She had seen the ships , but at such a great distance that they looked like sea-gulls . The dolphins sported in the waves , and the great whales spouted water from their nostrils till it seemed as if a hundred fountains were playing in every direc
 tion . The fifth sister 's birthday occurred in the winter ; so when her turn came , she saw what the others had not seen the first time they went up . The sea looked quite green , and large icebergs were floating about , each like a pearl , she said , but larger and loftier than the churches built by men . They were of the most singular shapes , and glittered like diamonds . She had seated herself upon one of the largest , and let the wind play with her long hair , and she remarked that all the ships sailed by rapidly , and steered as far away as they could from the iceberg , as if they were afraid of it . Towards evening , as the sun went down , dark clouds covered the sky , the thunder rolled and the lightning flashed , and the red light glowed on the icebergs as they rocked and tossed on the heaving sea . On all the ships the sails were reefed with fear and trembling , while she sat calmly on the floating iceberg , watching the blue lightning , as it darted its forked flashes in
 to the sea . When first the sisters had permission to rise to the surface , they were each delighted with the new and beautiful sights they saw ; but now , as grown-up girls , they could go when they pleased , and they had become indifferent about it . They wished themselves back again in the water , and after a month had passed they said it was much more beautiful down below , and pleasanter to be at home . Yet often , in the evening hours , the five sisters would twine their arms round each other , and rise to the surface , in a row . They had more beautiful voices than any human being could have ; and before the approach of a storm , and when they expected a ship would be lost , they swam before the vessel , and sang sweetly of the delights to be found in the depths of the sea , and begging the sailors not to fear if they sank to the bottom . But the sailors could not understand the song , they took it for the howling of the storm . And these things were never to be beautiful for
  them ; for if the ship sank , the men were drowned , and their dead bodies alone reached the palace of the Sea King . When the sisters rose , arm-in-arm , through the water in this way , their youngest sister would stand quite alone , looking after them , ready to cry , only that the mermaids have no tears , and therefore they suffer more . \u201c Oh , were I but fifteen years old , \u201d said she : \u201c I know that I shall love the world up there , and all the people who live in it . \u201d At last she reached her fifteenth year . \u201c Well , now , you are grown up , \u201d said the old dowager , her grandmother ; \u201c so you must let me adorn you like your other sisters ; \u201d and she placed a wreath of white lilies in her hair , and every flower leaf was half a pearl . Then the old lady ordered eight great oysters to attach themselves to the tail of the princess to show her high rank . \u201c But they hurt me so , \u201d said the little mermaid . \u201c Pride must suffer pain , \u201d replied the old lady
  . Oh , how gladly she would have shaken off all this grandeur , and laid aside the heavy wreath ! The red flowers in her own garden would have suited her much better , but she could not help herself : so she said , \u201c Farewell , \u201d and rose as lightly as a bubble to the surface of the water . The sun had just set as she raised her head above the waves ; but the clouds were tinted with crimson and gold , and through the glimmering twilight beamed the evening star in all its beauty . The sea was calm , and the air mild and fresh . A large ship , with three masts , lay becalmed on the water , with only one sail set ; for not a breeze stiffed , and the sailors sat idle on deck or amongst the rigging . There was music and song on board ; and , as darkness came on , a hundred colored lanterns were lighted , as if the flags of all nations waved in the air . The little mermaid swam close to the cabin windows ; and now and then , as the waves lifted her up , she could look in through clea
 r glass window-panes , and see a number of well-dressed people within . Among them was a young prince , the most beautiful of all , with large black eyes ; he was sixteen years of age , and his birthday was being kept with much rejoicing . The sailors were dancing on deck , but when the prince came out of the cabin , more than a hundred rockets rose in the air , making it as bright as day . The little mermaid was so startled that she dived under water ; and when she again stretched out her head , it appeared as if all the stars of heaven were falling around her , she had never seen such fireworks before . Great suns spurted fire about , splendid fireflies flew into the blue air , and everything was reflected in the clear , calm sea beneath . The ship itself was so brightly illuminated that all the people , and even the smallest rope , could be distinctly and plainly seen . And how handsome the young prince looked , as he pressed the hands of all present and smiled at them , while th
 e music resounded through the clear night air . It was very late ; yet the little mermaid could not take her eyes from the ship , or from the beautiful prince . The colored lanterns had been extinguished , no more rockets rose in the air , and the cannon had ceased firing ; but the sea became restless , and a moaning , grumbling sound could be heard beneath the waves : still the little mermaid remained by the cabin window , rocking up and down on the water , which enabled her to look in . After a while , the sails were quickly unfurled , and the noble ship continued her passage ; but soon the waves rose higher , heavy clouds darkened the sky , and lightning appeared in the distance . A dreadful storm was approaching ; once more the sails were reefed , and the great ship pursued her flying course over the raging sea . The waves rose mountains high , as if they would have overtopped the mast ; but the ship dived like a swan between them , and then rose again on their lofty , foaming c
 rests . To the little mermaid this appeared pleasant sport ; not so to the sailors . At length the ship groaned and creaked ; the thick planks gave way under the lashing of the sea as it broke over the deck ; the mainmast snapped asunder like a reed ; the ship lay over on her side ; and the water rushed in . The little mermaid now perceived that the crew were in danger ; even she herself was obliged to be careful to avoid the beams and planks of the wreck which lay scattered on the water . At one moment it was so pitch dark that she could not see a single object , but a flash of lightning revealed the whole scene ; she could see every one who had been on board excepting the prince ; when the ship parted , she had seen him sink into the deep waves , and she was glad , for she thought he would now be with her ; and then she remembered that human beings could not live in the water , so that when he got down to her father 's palace he would be quite dead . But he must not die . So she s
 wam about among the beams and planks which strewed the surface of the sea , forgetting that they could crush her to pieces . Then she dived deeply under the dark waters , rising and falling with the waves , till at length she managed to reach the young prince , who was fast losing the power of swimming in that stormy sea . His limbs were failing him , his beautiful eyes were closed , and he would have died had not the little mermaid come to his assistance . She held his head above the water , and let the waves drift them where they would . In the morning the storm had ceased ; but of the ship not a single fragment could be seen . The sun rose up red and glowing from the water , and its beams brought back the hue of health to the prince 's cheeks ; but his eyes remained closed . The mermaid kissed his high , smooth forehead , and stroked back his wet hair ; he seemed to her like the marble statue in her little garden , and she kissed him again , and wished that he might live . Presen
 tly they came in sight of land ; she saw lofty blue mountains , on which the white snow rested as if a flock of swans were lying upon them . Near the coast were beautiful green forests , and close by stood a large building , whether a church or a convent she could not tell . Orange and citron trees grew in the garden , and before the door stood lofty palms . The sea here formed a little bay , in which the water was quite still , but very deep ; so she swam with the handsome prince to the beach , which was covered with fine , white sand , and there she laid him in the warm sunshine , taking care to raise his head higher than his body . Then bells sounded in the large white building , and a number of young girls came into the garden . The little mermaid swam out farther from the shore and placed herself between some high rocks that rose out of the water ; then she covered her head and neck with the foam of the sea so that her little face might not be seen , and watched to see what wou
 ld become of the poor prince . She did not wait long before she saw a young girl approach the spot where he lay . She seemed frightened at first , but only for a moment ; then she fetched a number of people , and the mermaid saw that the prince came to life again , and smiled upon those who stood round him . But to her he sent no smile ; he knew not that she had saved him . This made her very unhappy , and when he was led away into the great building , she dived down sorrowfully into the water , and returned to her father 's castle . She had always been silent and thoughtful , and now she was more so than ever . Her sisters asked her what she had seen during her first visit to the surface of the water ; but she would tell them nothing . Many an evening and morning did she rise to the place where she had left the prince . She saw the fruits in the garden ripen till they were gathered , the snow on the tops of the mountains melt away ; but she never saw the prince , and therefore she 
 returned home , always more sorrowful than before . It was her only comfort to sit in her own little garden , and fling her arm round the beautiful marble statue which was like the prince ; but she gave up tending her flowers , and they grew in wild confusion over the paths , twining their long leaves and stems round the branches of the trees , so that the whole place became dark and gloomy . At length she could bear it no longer , and told one of her sisters all about it . Then the others heard the secret , and very soon it became known to two mermaids whose intimate friend happened to know who the prince was . She had also seen the festival on board ship , and she told them where the prince came from , and where his palace stood . \u201c Come , little sister , \u201d said the other princesses ; then they entwined their arms and rose up in a long row to the surface of the water , close by the spot where they knew the prince 's palace stood . It was built of bright yellow shining stone , 
 with long flights of marble steps , one of which reached quite down to the sea . Splendid gilded cupolas rose over the roof , and between the pillars that surrounded the whole building stood life-like statues of marble . Through the clear crystal of the lofty windows could be seen noble rooms , with costly silk curtains and hangings of tapestry ; while the walls were covered with beautiful paintings which were a pleasure to look at . In the centre of the largest saloon a fountain threw its sparkling jets high up into the glass cupola of the ceiling , through which the sun shone down upon the water and upon the beautiful plants growing round the basin of the fountain . Now that she knew where he lived , she spent many an evening and many a night on the water near the palace . She would swim much nearer the shore than any of the others ventured to do ; indeed once she went quite up the narrow channel under the marble balcony , which threw a broad shadow on the water . Here she would s
 it and watch the young prince , who thought himself quite alone in the bright moonlight . She saw him many times of an evening sailing in a pleasant boat , with music playing and flags waving . She peeped out from among the green rushes , and if the wind caught her long silvery-white veil , those who saw it believed it to be a swan , spreading out its wings . On many a night , too , when the fishermen , with their torches , were out at sea , she heard them relate so many good things about the doings of the young prince , that she was glad she had saved his life when he had been tossed about half-dead on the waves . And she remembered that his head had rested on her bosom , and how heartily she had kissed him ; but he knew nothing of all this , and could not even dream of her . She grew more and more fond of human beings , and wished more and more to be able to wander about with those whose world seemed to be so much larger than her own . They could fly over the sea in ships , and mo
 unt the high hills which were far above the clouds ; and the lands they possessed , their woods and their fields , stretched far away beyond the reach of her sight . There was so much that she wished to know , and her sisters were unable to answer all her questions . Then she applied to her old grandmother , who knew all about the upper world , which she very rightly called the lands above the sea . \u201c If human beings are not drowned , \u201d asked the little mermaid , \u201c can they live forever ? do they never die as we do here in the sea ? \u201d \u201c Yes , \u201d replied the old lady , \u201c they must also die , and their term of life is even shorter than ours . We sometimes live to three hundred years , but when we cease to exist here we only become the foam on the surface of the water , and we have not even a grave down here of those we love . We have not immortal souls , we shall never live again ; but , like the green sea-weed , when once it has been cut off , we can never flourish more 
 . Human beings , on the contrary , have a soul which lives forever , lives after the body has been turned to dust . It rises up through the clear , pure air beyond the glittering stars . As we rise out of the water , and behold all the land of the earth , so do they rise to unknown and glorious regions which we shall never see . \u201d \u201c Why have not we an immortal soul ? \u201d asked the little mermaid mournfully ; \u201c I would give gladly all the hundreds of years that I have to live , to be a human being only for one day , and to have the hope of knowing the happiness of that glorious world above the stars . \u201d \u201c You must not think of that , \u201d said the old woman ; \u201c we feel ourselves to be much happier and much better off than human beings . \u201d \u201c So I shall die , \u201d said the little mermaid , \u201c and as the foam of the sea I shall be driven about never again to hear the music of the waves , or to see the pretty flowers nor the red sun . Is there anything I can do to win an i
 mmortal soul ? \u201d \u201c No , \u201d said the old woman , \u201c unless a man were to love you so much that you were more to him than his father or mother ; and if all his thoughts and all his love were fixed upon you , and the priest placed his right hand in yours , and he promised to be true to you here and hereafter , then his soul would glide into your body and you would obtain a share in the future happiness of mankind . He would give a soul to you and retain his own as well ; but this can never happen . Your fish 's tail , which amongst us is considered so beautiful , is thought on earth to be quite ugly ; they do not know any better , and they think it necessary to have two stout props , which they call legs , in order to be handsome . \u201d Then the little mermaid sighed , and looked sorrowfully at her fish 's tail . \u201c Let us be happy , \u201d said the old lady , \u201c and dart and spring about during the three hundred years that we have to live , which is really quite long enough ; afte
 r that we can rest ourselves all the better . This evening we are going to have a court ball . \u201d It is one of those splendid sights which we can never see on earth . The walls and the ceiling of the large ball-room were of thick , but transparent crystal . May hundreds of colossal shells , some of a deep red , others of a grass green , stood on each side in rows , with blue fire in them , which lighted up the whole saloon , and shone through the walls , so that the sea was also illuminated . Innumerable fishes , great and small , swam past the crystal walls ; on some of them the scales glowed with a purple brilliancy , and on others they shone like silver and gold . Through the halls flowed a broad stream , and in it danced the mermen and the mermaids to the music of their own sweet singing . No one on earth has such a lovely voice as theirs . The little mermaid sang more sweetly than them all . The whole court applauded her with hands and tails ; and for a moment her heart felt q
 uite gay , for she knew she had the loveliest voice of any on earth or in the sea . But she soon thought again of the world above her , for she could not forget the charming prince , nor her sorrow that she had not an immortal soul like his ; therefore she crept away silently out of her father 's palace , and while everything within was gladness and song , she sat in her own little garden sorrowful and alone . Then she heard the bugle sounding through the water , and thought \u2013 \u201c He is certainly sailing above , he on whom my wishes depend , and in whose hands I should like to place the happiness of my life . I will venture all for him , and to win an immortal soul , while my sisters are dancing in my father 's palace , I will go to the sea witch , of whom I have always been so much afraid , but she can give me counsel and help . \u201d And then the little mermaid went out from her garden , and took the road to the foaming whirlpools , behind which the sorceress lived . She had never
  been that way before : neither flowers nor grass grew there ; nothing but bare , gray , sandy ground stretched out to the whirlpool , where the water , like foaming mill-wheels , whirled round everything that it seized , and cast it into the fathomless deep . Through the midst of these crushing whirlpools the little mermaid was obliged to pass , to reach the dominions of the sea witch ; and also for a long distance the only road lay right across a quantity of warm , bubbling mire , called by the witch her turfmoor . Beyond this stood her house , in the centre of a strange forest , in which all the trees and flowers were polypi , half animals and half plants ; they looked like serpents with a hundred heads growing out of the ground . The branches were long slimy arms , with fingers like flexible worms , moving limb after limb from the root to the top . All that could be reached in the sea they seized upon , and held fast , so that it never escaped from their clutches . The little me
 rmaid was so alarmed at what she saw , that she stood still , and her heart beat with fear , and she was very nearly turning back ; but she thought of the prince , and of the human soul for which she longed , and her courage returned . She fastened her long flowing hair round her head , so that the polypi might not seize hold of it . She laid her hands together across her bosom , and then she darted forward as a fish shoots through the water , between the supple arms and fingers of the ugly polypi , which were stretched out on each side of her . She saw that each held in its grasp something it had seized with its numerous little arms , as if they were iron bands . The white skeletons of human beings who had perished at sea , and had sunk down into the deep waters , skeletons of land animals , oars , rudders , and chests of ships were lying tightly grasped by their clinging arms ; even a little mermaid , whom they had caught and strangled ; and this seemed the most shocking of all to
  the little princess . She now came to a space of marshy ground in the wood , where large , fat water-snakes were rolling in the mire , and showing their ugly , drab-colored bodies . In the midst of this spot stood a house , built with the bones of shipwrecked human beings . There sat the sea witch , allowing a toad to eat from her mouth , just as people sometimes feed a canary with a piece of sugar . She called the ugly water-snakes her little chickens , and allowed them to crawl all over her bosom . \u201c I know what you want , \u201d said the sea witch ; \u201c it is very stupid of you , but you shall have your way , and it will bring you to sorrow , my pretty princess . You want to get rid of your fish 's tail , and to have two supports instead of it , like human beings on earth , so that the young prince may fall in love with you , and that you may have an immortal soul . \u201d And then the witch laughed so loud and disgustingly , that the toad and the snakes fell to the ground , and lay
  there wriggling about . \u201c You are but just in time , \u201d said the witch ; \u201c for after sunrise to-morrow I should not be able to help you till the end of another year . I will prepare a draught for you , with which you must swim to land tomorrow before sunrise , and sit down on the shore and drink it . Your tail will then disappear , and shrink up into what mankind calls legs , and you will feel great pain , as if a sword were passing through you . But all who see you will say that you are the prettiest little human being they ever saw . You will still have the same floating gracefulness of movement , and no dancer will ever tread so lightly ; but at every step you take it will feel as if you were treading upon sharp knives , and that the blood must flow . If you will bear all this , I will help you . \u201d \u201c Yes , I will , \u201d said the little princess in a trembling voice , as she thought of the prince and the immortal soul . \u201c But think again , \u201d said the witch ; \u201c for w
 hen once your shape has become like a human being , you can no more be a mermaid . You will never return through the water to your sisters , or to your father 's palace again ; and if you do not win the love of the prince , so that he is willing to forget his father and mother for your sake , and to love you with his whole soul , and allow the priest to join your hands that you may be man and wife , then you will never have an immortal soul . The first morning after he marries another your heart will break , and you will become foam on the crest of the waves . \u201d \u201c I will do it , \u201d said the little mermaid , and she became pale as death . \u201c But I must be paid also , \u201d said the witch , \u201c and it is not a trifle that I ask . You have the sweetest voice of any who dwell here in the depths of the sea , and you believe that you will be able to charm the prince with it also , but this voice you must give to me ; the best thing you possess will I have for the price of my draught .
  My own blood must be mixed with it , that it may be as sharp as a two-edged sword . \u201d \u201c But if you take away my voice , \u201d said the little mermaid , \u201c what is left for me ? \u201d \u201c Your beautiful form , your graceful walk , and your expressive eyes ; surely with these you can enchain a man 's heart . Well , have you lost your courage ? Put out your little tongue that I may cut it off as my payment ; then you shall have the powerful draught . \u201d \u201c It shall be , \u201d said the little mermaid . Then the witch placed her cauldron on the fire , to prepare the magic draught . \u201c Cleanliness is a good thing , \u201d said she , scouring the vessel with snakes , which she had tied together in a large knot ; then she pricked herself in the breast , and let the black blood drop into it . The steam that rose formed itself into such horrible shapes that no one could look at them without fear . Every moment the witch threw something else into the vessel , and when it began to boil , the so
 und was like the weeping of a crocodile . When at last the magic draught was ready , it looked like the clearest water . \u201c There it is for you , \u201d said the witch . Then she cut off the mermaid 's tongue , so that she became dumb , and would never again speak or sing . \u201c If the polypi should seize hold of you as you return through the wood , \u201d said the witch , \u201c throw over them a few drops of the potion , and their fingers will be torn into a thousand pieces . \u201d But the little mermaid had no occasion to do this , for the polypi sprang back in terror when they caught sight of the glittering draught , which shone in her hand like a twinkling star . So she passed quickly through the wood and the marsh , and between the rushing whirlpools . She saw that in her father 's palace the torches in the ballroom were extinguished , and all within asleep ; but she did not venture to go in to them , for now she was dumb and going to leave them forever , she felt as if her heart would b
 reak . She stole into the garden , took a flower from the flower-beds of each of her sisters , kissed her hand a thousand times towards the palace , and then rose up through the dark blue waters . The sun had not risen when she came in sight of the prince 's palace , and approached the beautiful marble steps , but the moon shone clear and bright . Then the little mermaid drank the magic draught , and it seemed as if a two-edged sword went through her delicate body : she fell into a swoon , and lay like one dead . When the sun arose and shone over the sea , she recovered , and felt a sharp pain ; but just before her stood the handsome young prince . He fixed his coal-black eyes upon her so earnestly that she cast down her own , and then became aware that her fish 's tail was gone , and that she had as pretty a pair of white legs and tiny feet as any little maiden could have ; but she had no clothes , so she wrapped herself in her long , thick hair . The prince asked her who she was ,
  and where she came from , and she looked at him mildly and sorrowfully with her deep blue eyes ; but she could not speak . Every step she took was as the witch had said it would be , she felt as if treading upon the points of needles or sharp knives ; but she bore it willingly , and stepped as lightly by the prince 's side as a soap-bubble , so that he and all who saw her wondered at her graceful-swaying movements . She was very soon arrayed in costly robes of silk and muslin , and was the most beautiful creature in the palace ; but she was dumb , and could neither speak nor sing . Beautiful female slaves , dressed in silk and gold , stepped forward and sang before the prince and his royal parents : one sang better than all the others , and the prince clapped his hands and smiled at her . This was great sorrow to the little mermaid ; she knew how much more sweetly she herself could sing once , and she thought , \u201c Oh if he could only know that ! I have given away my voice forever 
 , to be with him . \u201d The slaves next performed some pretty fairy-like dances , to the sound of beautiful music . Then the little mermaid raised her lovely white arms , stood on the tips of her toes , and glided over the floor , and danced as no one yet had been able to dance . At each moment her beauty became more revealed , and her expressive eyes appealed more directly to the heart than the songs of the slaves . Every one was enchanted , especially the prince , who called her his little foundling ; and she danced again quite readily , to please him , though each time her foot touched the floor it seemed as if she trod on sharp knives . The prince said she should remain with him always , and she received permission to sleep at his door , on a velvet cushion . He had a page 's dress made for her , that she might accompany him on horseback . They rode together through the sweet-scented woods , where the green boughs touched their shoulders , and the little birds sang among the fres
 h leaves . She climbed with the prince to the tops of high mountains ; and although her tender feet bled so that even her steps were marked , she only laughed , and followed him till they could see the clouds beneath them looking like a flock of birds travelling to distant lands . While at the prince 's palace , and when all the household were asleep , she would go and sit on the broad marble steps ; for it eased her burning feet to bathe them in the cold sea-water ; and then she thought of all those below in the deep . Once during the night her sisters came up arm-in-arm , singing sorrowfully , as they floated on the water . She beckoned to them , and then they recognized her , and told her how she had grieved them . After that , they came to the same place every night ; and once she saw in the distance her old grandmother , who had not been to the surface of the sea for many years , and the old Sea King , her father , with his crown on his head . They stretched out their hands tow
 ards her , but they did not venture so near the land as her sisters did . As the days passed , she loved the prince more fondly , and he loved her as he would love a little child , but it never came into his head to make her his wife ; yet , unless he married her , she could not receive an immortal soul ; and , on the morning after his marriage with another , she would dissolve into the foam of the sea . \u201c Do you not love me the best of them all ? \u201d the eyes of the little mermaid seemed to say , when he took her in his arms , and kissed her fair forehead . \u201c Yes , you are dear to me , \u201d said the prince ; \u201c for you have the best heart , and you are the most devoted to me ; you are like a young maiden whom I once saw , but whom I shall never meet again . I was in a ship that was wrecked , and the waves cast me ashore near a holy temple , where several young maidens performed the service . The youngest of them found me on the shore , and saved my life . I saw her but twice , 
 and she is the only one in the world whom I could love ; but you are like her , and you have almost driven her image out of my mind . She belongs to the holy temple , and my good fortune has sent you to me instead of her ; and we will never part . \u201d \u201c Ah , he knows not that it was I who saved his life , \u201d thought the little mermaid . \u201c I carried him over the sea to the wood where the temple stands : I sat beneath the foam , and watched till the human beings came to help him . I saw the pretty maiden that he loves better than he loves me ; \u201d and the mermaid sighed deeply , but she could not shed tears . \u201c He says the maiden belongs to the holy temple , therefore she will never return to the world . They will meet no more : while I am by his side , and see him every day . I will take care of him , and love him , and give up my life for his sake . \u201d Very soon it was said that the prince must marry , and that the beautiful daughter of a neighboring king would be his wife ,
  for a fine ship was being fitted out . Although the prince gave out that he merely intended to pay a visit to the king , it was generally supposed that he really went to see his daughter . A great company were to go with him . The little mermaid smiled , and shook her head . She knew the prince 's thoughts better than any of the others . \u201c I must travel , \u201d he had said to her ; \u201c I must see this beautiful princess ; my parents desire it ; but they will not oblige me to bring her home as my bride . I cannot love her ; she is not like the beautiful maiden in the temple , whom you resemble . If I were forced to choose a bride , I would rather choose you , my dumb foundling , with those expressive eyes . \u201d And then he kissed her rosy mouth , played with her long waving hair , and laid his head on her heart , while she dreamed of human happiness and an immortal soul . \u201c You are not afraid of the sea , my dumb child , \u201d said he , as they stood on the deck of the noble ship wh
 ich was to carry them to the country of the neighboring king . And then he told her of storm and of calm , of strange fishes in the deep beneath them , and of what the divers had seen there ; and she smiled at his descriptions , for she knew better than any one what wonders were at the bottom of the sea . In the moonlight , when all on board were asleep , excepting the man at the helm , who was steering , she sat on the deck , gazing down through the clear water . She thought she could distinguish her father 's castle , and upon it her aged grandmother , with the silver crown on her head , looking through the rushing tide at the keel of the vessel . Then her sisters came up on the waves , and gazed at her mournfully , wringing their white hands . She beckoned to them , and smiled , and wanted to tell them how happy and well off she was ; but the cabin-boy approached , and when her sisters dived down he thought it was only the foam of the sea which he saw . The next morning the ship 
 sailed into the harbor of a beautiful town belonging to the king whom the prince was going to visit . The church bells were ringing , and from the high towers sounded a flourish of trumpets ; and soldiers , with flying colors and glittering bayonets , lined the rocks through which they passed . Every day was a festival ; balls and entertainments followed one another . But the princess had not yet appeared . People said that she was being brought up and educated in a religious house , where she was learning every royal virtue . At last she came . Then the little mermaid , who was very anxious to see whether she was really beautiful , was obliged to acknowledge that she had never seen a more perfect vision of beauty . Her skin was delicately fair , and beneath her long dark eye-lashes her laughing blue eyes shone with truth and purity . \u201c It was you , \u201d said the prince , \u201c who saved my life when I lay dead on the beach , \u201d and he folded his blushing bride in his arms . \u201c Oh 
 , I am too happy , \u201d said he to the little mermaid ; \u201c my fondest hopes are all fulfilled . You will rejoice at my happiness ; for your devotion to me is great and sincere . \u201d The little mermaid kissed his hand , and felt as if her heart were already broken . His wedding morning would bring death to her , and she would change into the foam of the sea . All the church bells rung , and the heralds rode about the town proclaiming the betrothal . Perfumed oil was burning in costly silver lamps on every altar . The priests waved the censers , while the bride and bridegroom joined their hands and received the blessing of the bishop . The little mermaid , dressed in silk and gold , held up the bride 's train ; but her ears heard nothing of the festive music , and her eyes saw not the holy ceremony ; she thought of the night of death which was coming to her , and of all she had lost in the world . On the same evening the bride and bridegroom went on board ship ; cannons were roaring ,
  flags waving , and in the centre of the ship a costly tent of purple and gold had been erected . It contained elegant couches , for the reception of the bridal pair during the night . The ship , with swelling sails and a favorable wind , glided away smoothly and lightly over the calm sea . When it grew dark a number of colored lamps were lit , and the sailors danced merrily on the deck . The little mermaid could not help thinking of her first rising out of the sea , when she had seen similar festivities and joys ; and she joined in the dance , poised herself in the air as a swallow when he pursues his prey , and all present cheered her with wonder . She had never danced so elegantly before . Her tender feet felt as if cut with sharp knives , but she cared not for it ; a sharper pang had pierced through her heart . She knew this was the last evening she should ever see the prince , for whom she had forsaken her kindred and her home ; she had given up her beautiful voice , and suffer
 ed unheard-of pain daily for him , while he knew nothing of it . This was the last evening that she would breathe the same air with him , or gaze on the starry sky and the deep sea ; an eternal night , without a thought or a dream , awaited her : she had no soul and now she could never win one . All was joy and gayety on board ship till long after midnight ; she laughed and danced with the rest , while the thoughts of death were in her heart . The prince kissed his beautiful bride , while she played with his raven hair , till they went arm-in-arm to rest in the splendid tent . Then all became still on board the ship ; the helmsman , alone awake , stood at the helm . The little mermaid leaned her white arms on the edge of the vessel , and looked towards the east for the first blush of morning , for that first ray of dawn that would bring her death . She saw her sisters rising out of the flood : they were as pale as herself ; but their long beautiful hair waved no more in the wind , a
 nd had been cut off . \u201c We have given our hair to the witch , \u201d said they , \u201c to obtain help for you , that you may not die to-night . She has given us a knife : here it is , see it is very sharp . Before the sun rises you must plunge it into the heart of the prince ; when the warm blood falls upon your feet they will grow together again , and form into a fish 's tail , and you will be once more a mermaid , and return to us to live out your three hundred years before you die and change into the salt sea foam . Haste , then ; he or you must die before sunrise . Our old grandmother moans so for you , that her white hair is falling off from sorrow , as ours fell under the witch 's scissors . Kill the prince and come back ; hasten : do you not see the first red streaks in the sky ? In a few minutes the sun will rise , and you must die . \u201d And then they sighed deeply and mournfully , and sank down beneath the waves . The little mermaid drew back the crimson curtain of the tent , 
 and beheld the fair bride with her head resting on the prince 's breast . She bent down and kissed his fair brow , then looked at the sky on which the rosy dawn grew brighter and brighter ; then she glanced at the sharp knife , and again fixed her eyes on the prince , who whispered the name of his bride in his dreams . She was in his thoughts , and the knife trembled in the hand of the little mermaid : then she flung it far away from her into the waves ; the water turned red where it fell , and the drops that spurted up looked like blood . She cast one more lingering , half-fainting glance at the prince , and then threw herself from the ship into the sea , and thought her body was dissolving into foam . The sun rose above the waves , and his warm rays fell on the cold foam of the little mermaid , who did not feel as if she were dying . She saw the bright sun , and all around her floated hundreds of transparent beautiful beings ; she could see through them the white sails of the ship
  , and the red clouds in the sky ; their speech was melodious , but too ethereal to be heard by mortal ears , as they were also unseen by mortal eyes . The little mermaid perceived that she had a body like theirs , and that she continued to rise higher and higher out of the foam . \u201c Where am I ? \u201d asked she , and her voice sounded ethereal , as the voice of those who were with her ; no earthly music could imitate it . \u201c Among the daughters of the air , \u201d answered one of them . \u201c A mermaid has not an immortal soul , nor can she obtain one unless she wins the love of a human being . On the power of another hangs her eternal destiny . But the daughters of the air , although they do not possess an immortal soul , can , by their good deeds , procure one for themselves . We fly to warm countries , and cool the sultry air that destroys mankind with the pestilence . We carry the perfume of the flowers to spread health and restoration . After we have striven for three hundred years
  to all the good in our power , we receive an immortal soul and take part in the happiness of mankind . You , poor little mermaid , have tried with your whole heart to do as we are doing ; you have suffered and endured and raised yourself to the spirit-world by your good deeds ; and now , by striving for three hundred years in the same way , you may obtain an immortal soul . \u201d The little mermaid lifted her glorified eyes towards the sun , and felt them , for the first time , filling with tears . On the ship , in which she had left the prince , there were life and noise ; she saw him and his beautiful bride searching for her ; sorrowfully they gazed at the pearly foam , as if they knew she had thrown herself into the waves . Unseen she kissed the forehead of her bride , and fanned the prince , and then mounted with the other children of the air to a rosy cloud that floated through the aether . \u201c After three hundred years , thus shall we float into the kingdom of heaven , \u201d said
  she . \u201c And we may even get there sooner , \u201d whispered one of her companions . \u201c Unseen we can enter the houses of men , where there are children , and for every day on which we find a good child , who is the joy of his parents and deserves their love , our time of probation is shortened . The child does not know , when we fly through the room , that we smile with joy at his good conduct , for we can count one year less of our three hundred years . But when we see a naughty or a wicked child , we shed tears of sorrow , and for every tear a day is added to our time of trial ! \u201d 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/45FictAndersenH_Ugly_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/45FictAndersenH_Ugly_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/45FictAndersenH_Ugly_EN.txt.txt
new file mode 100644
index 0000000..ed3ae9f
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/45FictAndersenH_Ugly_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff The ugly duckling It was lovely summer weather in the country , and the golden corn , the green oats , and the haystacks piled up in the meadows looked beautiful . The stork walking about on his long red legs chattered in the Egyptian language which he had learnt from his mother . The corn-fields and meadows were surrounded by large forests , in the midst of which were deep pools . It was , indeed , delightful to walk about in the country . In a sunny spot stood a pleasant old farm-house close by a deep river , and from the house down to the water side grew great burdock leaves , so high , that under the tallest of them a little child could stand upright . The spot was as wild as the centre of a thick wood . In this snug retreat sat a duck on her nest , watching for her young brood to hatch ; she was beginning to get tired of her task , for the little ones were a long time coming out of their shells , and she seldom had any visitors . The other ducks liked much better to swim ab
 out in the river than to climb the slippery banks , and sit under a burdock leaf , to have a gossip with her . At length one shell cracked , and then another , and from each egg came a living creature that lifted its head and cried , \u201c Peep , peep . \u201d \u201c Quack , quack , \u201d said the mother , and then they all quacked as well as they could , and looked about them on every side at the large green leaves . Their mother allowed them to look as much as they liked , because green is good for the eyes . \u201c How large the world is , \u201d said the young ducks , when they found how much more room they now had than while they were inside the egg-shell . \u201c Do you imagine this is the whole world ? \u201d asked the mother ; \u201c Wait till you have seen the garden ; it stretches far beyond that to the parson 's field , but I have never ventured to such a distance . Are you all out ? \u201d she continued , rising ; \u201c No , I declare , the largest egg lies there still . I wonder how long this is to 
 last , I am quite tired of it ; \u201d and she seated herself again on the nest . \u201c Well , how are you getting on ? \u201d asked an old duck , who paid her a visit . \u201c One egg is not hatched yet , \u201d said the duck , \u201c it will not break . But just look at all the others , are they not the prettiest little ducklings you ever saw ? They are the image of their father , who is so unkind , he never comes to see . \u201d \u201c Let me see the egg that will not break , \u201d said the duck ; \u201c I have no doubt it is a turkey 's egg . I was persuaded to hatch some once , and after all my care and trouble with the young ones , they were afraid of the water . I quacked and clucked , but all to no purpose . I could not get them to venture in . Let me look at the egg . Yes , that is a turkey 's egg ; take my advice , leave it where it is and teach the other children to swim . \u201d \u201c I think I will sit on it a little while longer , \u201d said the duck ; \u201c as I have sat so long already , a few days will
  be nothing . \u201d \u201c Please yourself , \u201d said the old duck , and she went away . At last the large egg broke , and a young one crept forth crying , \u201c Peep , peep . \u201d It was very large and ugly . The duck stared at it and exclaimed , \u201c It is very large and not at all like the others . I wonder if it really is a turkey . We shall soon find it out , however when we go to the water . It must go in , if I have to push it myself . \u201d On the next day the weather was delightful , and the sun shone brightly on the green burdock leaves , so the mother duck took her young brood down to the water , and jumped in with a splash . \u201c Quack , quack , \u201d cried she , and one after another the little ducklings jumped in . The water closed over their heads , but they came up again in an instant , and swam about quite prettily with their legs paddling under them as easily as possible , and the ugly duckling was also in the water swimming with them . \u201c Oh , \u201d said the mother , \u201c that is
  not a turkey ; how well he uses his legs , and how upright he holds himself ! He is my own child , and he is not so very ugly after all if you look at him properly . Quack , quack ! come with me now , I will take you into grand society , and introduce you to the farmyard , but you must keep close to me or you may be trodden upon ; and , above all , beware of the cat . \u201d When they reached the farmyard , there was a great disturbance , two families were fighting for an eel 's head , which , after all , was carried off by the cat . \u201c See , children , that is the way of the world , \u201d said the mother duck , whetting her beak , for she would have liked the eel 's head herself . \u201c Come , now , use your legs , and let me see how well you can behave . You must bow your heads prettily to that old duck yonder ; she is the highest born of them all , and has Spanish blood , therefore , she is well off . Do n't you see she has a red flag tied to her leg , which is something very grand , 
 and a great honor for a duck ; it shows that every one is anxious not to lose her , as she can be recognized both by man and beast . Come , now , do n't turn your toes , a well-bred duckling spreads his feet wide apart , just like his father and mother , in this way ; now bend your neck , and say \u2018 quack . \u2019 \u201d The ducklings did as they were bid , but the other duck stared , and said , \u201c Look , here comes another brood , as if there were not enough of us already ! and what a queer looking object one of them is ; we do n't want him here , \u201d and then one flew out and bit him in the neck . \u201c Let him alone , \u201d said the mother ; \u201c he is not doing any harm . \u201d \u201c Yes , but he is so big and ugly , \u201d said the spiteful duck \u201c and therefore he must be turned out . \u201d \u201c The others are very pretty children , \u201d said the old duck , with the rag on her leg , \u201c all but that one ; I wish his mother could improve him a little . \u201d \u201c That is impossible , your grace , 
  replied the mother ; \u201c he is not pretty ; but he has a very good disposition , and swims as well or even better than the others . I think he will grow up pretty , and perhaps be smaller ; he has remained too long in the egg , and therefore his figure is not properly formed ; \u201d and then she stroked his neck and smoothed the feathers , saying , \u201c It is a drake , and therefore not of so much consequence . I think he will grow up strong , and able to take care of himself . \u201d \u201c The other ducklings are graceful enough , \u201d said the old duck . \u201c Now make yourself at home , and if you can find an eel 's head , you can bring it to me . \u201d And so they made themselves comfortable . But the poor duckling , who had crept out of his shell last of all , and looked so ugly , was bitten and pushed and made fun of , not only by the ducks , but by all the poultry . \u201c He is too big , \u201d they all said , and the turkey cock , who had been born into the world with spurs , and fancied hi
 mself really an emperor , puffed himself out like a vessel in full sail , and flew at the duckling , and became quite red in the head with passion , so that the poor little thing did not know where to go , and was quite miserable because he was so ugly and laughed at by the whole farmyard . So it went on from day to day till it got worse and worse . The poor duckling was driven about by every one ; even his brothers and sisters were unkind to him , and would say , \u201c Ah , you ugly creature , I wish the cat would get you , \u201d and his mother said she wished he had never been born . The ducks pecked him , the chickens beat him , and the girl who fed the poultry kicked him with her feet . So at last he ran away , frightening the little birds in the hedge as he flew over the palings . \u201c They are afraid of me because I am ugly , \u201d he said . So he closed his eyes , and flew still farther , until he came out on a large moor , inhabited by wild ducks . Here he remained the whole night 
 , feeling very tired and sorrowful . In the morning , when the wild ducks rose in the air , they stared at their new comrade . \u201c What sort of a duck are you ? \u201d they all said , coming round him . He bowed to them , and was as polite as he could be , but he did not reply to their question . \u201c You are exceedingly ugly , \u201d said the wild ducks , \u201c but that will not matter if you do not want to marry one of our family . \u201d Poor thing ! he had no thoughts of marriage ; all he wanted was permission to lie among the rushes , and drink some of the water on the moor . After he had been on the moor two days , there came two wild geese , or rather goslings , for they had not been out of the egg long , and were very saucy . \u201c Listen , friend , \u201d said one of them to the duckling , \u201c you are so ugly , that we like you very well . Will you go with us , and become a bird of passage ? Not far from here is another moor , in which there are some pretty wild geese , all unmarried . It i
 s a chance for you to get a wife ; you may be lucky , ugly as you are . \u201d \u201c Pop , pop , \u201d sounded in the air , and the two wild geese fell dead among the rushes , and the water was tinged with blood . \u201c Pop , pop , \u201d echoed far and wide in the distance , and whole flocks of wild geese rose up from the rushes . The sound continued from every direction , for the sportsmen surrounded the moor , and some were even seated on branches of trees , overlooking the rushes . The blue smoke from the guns rose like clouds over the dark trees , and as it floated away across the water , a number of sporting dogs bounded in among the rushes , which bent beneath them wherever they went . How they terrified the poor duckling ! He turned away his head to hide it under his wing , and at the same moment a large terrible dog passed quite near him . His jaws were open , his tongue hung from his mouth , and his eyes glared fearfully . He thrust his nose close to the duckling , showing his sharp t
 eeth , and then , \u201c splash , splash , \u201d he went into the water without touching him . \u201c Oh , \u201d sighed the duckling , \u201c how thankful I am for being so ugly ; even a dog will not bite me . \u201d And so he lay quite still , while the shot rattled through the rushes , and gun after gun was fired over him . It was late in the day before all became quiet , but even then the poor young thing did not dare to move . He waited quietly for several hours , and then , after looking carefully around him , hastened away from the moor as fast as he could . He ran over field and meadow till a storm arose , and he could hardly struggle against it . Towards evening , he reached a poor little cottage that seemed ready to fall , and only remained standing because it could not decide on which side to fall first . The storm continued so violent , that the duckling could go no farther ; he sat down by the cottage , and then he noticed that the door was not quite closed in consequence of one of the 
 hinges having given way . There was therefore a narrow opening near the bottom large enough for him to slip through , which he did very quietly , and got a shelter for the night . A woman , a tom cat , and a hen lived in this cottage . The tom cat , whom the mistress called , \u201c My little son , \u201d was a great favorite ; he could raise his back , and purr , and could even throw out sparks from his fur if it were stroked the wrong way . The hen had very short legs , so she was called \u201c Chickie short legs . \u201d She laid good eggs , and her mistress loved her as if she had been her own child . In the morning , the strange visitor was discovered , and the tom cat began to purr , and the hen to cluck . \u201c What is that noise about ? \u201d said the old woman , looking round the room , but her sight was not very good ; therefore , when she saw the duckling she thought it must be a fat duck , that had strayed from home . \u201c Oh what a prize ! \u201d she exclaimed , \u201c I hope it is not a drak
 e , for then I shall have some duck 's eggs . I must wait and see . \u201d So the duckling was allowed to remain on trial for three weeks , but there were no eggs . Now the tom cat was the master of the house , and the hen was mistress , and they always said , \u201c We and the world , \u201d for they believed themselves to be half the world , and the better half too . The duckling thought that others might hold a different opinion on the subject , but the hen would not listen to such doubts . \u201c Can you lay eggs ? \u201d she asked . \u201c No. \u201d \u201c Then have the goodness to hold your tongue . \u201d \u201c Can you raise your back , or purr , or throw out sparks ? \u201d said the tom cat . \u201c No. \u201d \u201c Then you have no right to express an opinion when sensible people are speaking . \u201d So the duckling sat in a corner , feeling very low spirited , till the sunshine and the fresh air came into the room through the open door , and then he began to feel such a great longing for a swim on the water , tha
 t he could not help telling the hen . \u201c What an absurd idea , \u201d said the hen . \u201c You have nothing else to do , therefore you have foolish fancies . If you could purr or lay eggs , they would pass away . \u201d \u201c But it is so delightful to swim about on the water , \u201d said the duckling , \u201c and so refreshing to feel it close over your head , while you dive down to the bottom . \u201d \u201c Delightful , indeed ! \u201d said the hen , \u201c why you must be crazy ! Ask the cat , he is the cleverest animal I know , ask him how he would like to swim about on the water , or to dive under it , for I will not speak of my own opinion ; ask our mistress , the old woman \u2013 there is no one in the world more clever than she is . Do you think she would like to swim , or to let the water close over her head ? \u201d \u201c You do n't understand me , \u201d said the duckling . \u201c We do n't understand you ? Who can understand you , I wonder ? Do you consider yourself more clever than the cat , or the old woman
  ? I will say nothing of myself . Do n't imagine such nonsense , child , and thank your good fortune that you have been received here . Are you not in a warm room , and in society from which you may learn something . But you are a chatterer , and your company is not very agreeable . Believe me , I speak only for your own good . I may tell you unpleasant truths , but that is a proof of my friendship . I advise you , therefore , to lay eggs , and learn to purr as quickly as possible . \u201d \u201c I believe I must go out into the world again , \u201d said the duckling . \u201c Yes , do , \u201d said the hen . So the duckling left the cottage , and soon found water on which it could swim and dive , but was avoided by all other animals , because of its ugly appearance . Autumn came , and the leaves in the forest turned to orange and gold . Then , as winter approached , the wind caught them as they fell and whirled them in the cold air . The clouds , heavy with hail and snow-flakes , hung low in the s
 ky , and the raven stood on the ferns crying , \u201c Croak , croak . \u201d It made one shiver with cold to look at him . All this was very sad for the poor little duckling . One evening , just as the sun set amid radiant clouds , there came a large flock of beautiful birds out of the bushes . The duckling had never seen any like them before . They were swans , and they curved their graceful necks , while their soft plumage shown with dazzling whiteness . They uttered a singular cry , as they spread their glorious wings and flew away from those cold regions to warmer countries across the sea . As they mounted higher and higher in the air , the ugly little duckling felt quite a strange sensation as he watched them . He whirled himself in the water like a wheel , stretched out his neck towards them , and uttered a cry so strange that it frightened himself . Could he ever forget those beautiful , happy birds ; and when at last they were out of his sight , he dived under the water , and rose
  again almost beside himself with excitement . He knew not the names of these birds , nor where they had flown , but he felt towards them as he had never felt for any other bird in the world . He was not envious of these beautiful creatures , but wished to be as lovely as they . Poor ugly creature , how gladly he would have lived even with the ducks had they only given him encouragement . The winter grew colder and colder ; he was obliged to swim about on the water to keep it from freezing , but every night the space on which he swam became smaller and smaller . At length it froze so hard that the ice in the water crackled as he moved , and the duckling had to paddle with his legs as well as he could , to keep the space from closing up . He became exhausted at last , and lay still and helpless , frozen fast in the ice . Early in the morning , a peasant , who was passing by , saw what had happened . He broke the ice in pieces with his wooden shoe , and carried the duckling home to hi
 s wife . The warmth revived the poor little creature . But when the children wanted to play with him , the duckling thought they would do him some harm ; so he started up in terror , fluttered into the milk-pan , and splashed the milk about the room . Then the woman clapped her hands , which frightened him still more . He flew first into the butter-cask , then into the meal-tub , and out again . What a condition he was in ! The woman screamed , and struck at him with the tongs ; the children laughed and screamed , and tumbled over each other , in their efforts to catch him ; but luckily he escaped . The door stood open ; the poor creature could just manage to slip out among the bushes , and lie down quite exhausted in the newly fallen snow . It would be very sad , were I to relate all the misery and privations which the poor little duckling endured during the hard winter ; but when it had passed , he found himself lying one morning in a moor , amongst the rushes . He felt the warm s
 un shining , and heard the lark singing , and saw that all around was beautiful spring . Then the young bird felt that his wings were strong , as he flapped them against his sides , and rose high into the air . They bore him onwards , until he found himself in a large garden , before he well knew how it had happened . The apple-trees were in full blossom , and the fragrant elders bent their long green branches down to the stream which wound round a smooth lawn . Everything looked beautiful , in the freshness of early spring . From a thicket close by came three beautiful white swans , rustling their feathers , and swimming lightly over the smooth water . The duckling remembered the lovely birds , and felt more strangely unhappy than ever . \u201c I will fly to those royal birds , \u201d he exclaimed , \u201c and they will kill me , because I am so ugly , and dare to approach them ; but it does not matter : better be killed by them than pecked by the ducks , beaten by the hens , pushed about b
 y the maiden who feeds the poultry , or starved with hunger in the winter . \u201d Then he flew to the water , and swam towards the beautiful swans . The moment they espied the stranger , they rushed to meet him with outstretched wings . \u201c Kill me , \u201d said the poor bird ; and he bent his head down to the surface of the water , and awaited death . But what did he see in the clear stream below ? His own image ; no longer a dark , gray bird , ugly and disagreeable to look at , but a graceful and beautiful swan . To be born in a duck 's nest , in a farmyard , is of no consequence to a bird , if it is hatched from a swan 's egg . He now felt glad at having suffered sorrow and trouble , because it enabled him to enjoy so much better all the pleasure and happiness around him ; for the great swans swam round the new-comer , and stroked his neck with their beaks , as a welcome . Into the garden presently came some little children , and threw bread and cake into the water . \u201c See , \u201d cri
 ed the youngest , \u201c there is a new one ; \u201d and the rest were delighted , and ran to their father and mother , dancing and clapping their hands , and shouting joyously , \u201c There is another swan come ; a new one has arrived . \u201d Then they threw more bread and cake into the water , and said , \u201c The new one is the most beautiful of all ; he is so young and pretty . \u201d And the old swans bowed their heads before him . Then he felt quite ashamed , and hid his head under his wing ; for he did not know what to do , he was so happy , and yet not at all proud . He had been persecuted and despised for his ugliness , and now he heard them say he was the most beautiful of all the birds . Even the elder-tree bent down its bows into the water before him , and the sun shone warm and bright . Then he rustled his feathers , curved his slender neck , and cried joyfully , from the depths of his heart , \u201c I never dreamed of such happiness as this , while I was an ugly duckling . \u201d 
\ No newline at end of file


[13/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/140ted_a_j_jacobs_year_of_living_biblically.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/140ted_a_j_jacobs_year_of_living_biblically.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/140ted_a_j_jacobs_year_of_living_biblically.txt
new file mode 100644
index 0000000..1dd1e0f
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/140ted_a_j_jacobs_year_of_living_biblically.txt
@@ -0,0 +1,2 @@
+
+I thought I 'd tell you a little about what I like to write . And I like to immerse myself in my topics . I just like to dive right in and become sort of a human guinea pig . And I see my life as a series of experiments . So , I work for Esquire magazine , and a couple of years ago I wrote an article called " My Outsourced Life , " where I hired a team of people in Bangalore , India , to live my life for me . So they answered my emails . They answered my phone . They argued with my wife for me , and they read my son bedtime stories . It was the best month of my life , because I just sat back and I read books and watched movies . It was a wonderful experience . More recently , I wrote an article for Esquire called -- about radical honesty . And this is a movement -- this is started by a psychologist in Virginia , who says that you should never , ever lie , except maybe during poker and golf , his only exceptions . And , more than that , whatever is on your brain should come out of yo
 ur mouth . I decided I would try this for a month . This was the worst month of my life . ( Laughter ) I do not recommend this at all . To give you a sense of the experience , the article was called , " I Think You 're Fat . " ( Laughter ) So , that was hard . My most recent book -- my previous book was called " The Know-it-All , " and it was about the year I spent reading the Encyclopedia Britannica from A to Z in my quest to learn everything in the world , or more precisely from A-ak , which is a type of East Asian music , all the way to Zwyiec , which is -- well , I do n't want to ruin the ending . ( Laughter ) It 's a very exciting twist ending , like an O. Henry novel , so I wo n't ruin it . But I love that one because that was an experiment about how much information one human brain could absorb , although , listening to Kevin Kelly , you do n't have to remember anything . You can just Google it . So I wasted some time there . I love those experiments , but I think that the mo
 st profound and life-changing experiment that I 've done is my most recent experiment , where I spent a year trying to follow all of the rules of the Bible -- " The Year of Living Biblically . " And I undertook this for two reasons . The first was that I grew up with no religion at all . As I say in my book , I 'm Jewish in the same way the Olive Garden is Italian . ( Laughter ) So , not very . But I 've become increasingly interested in religion . I do think it 's the defining issue of our time , or one of the main ones . And I have a son . I want to know what to teach him . So , I decided to dive in head first , and try to live the Bible . The second reason I undertook this is because I 'm concerned about the rise of fundamentalism , religious fundamentalism , and people who say they take the Bible literally , which is , according to some polls , as high as 45 or 50 percent of America . So I decided , what if you really did take the Bible literally ? I decided to take it to its lo
 gical conclusion and take everything in the Bible literally , without picking and choosing . The first thing I did was I got a stack of bibles . I had Christian bibles . I had Jewish bibles . A friend of mine sent me something called a hip-hop bible , where the 23rd Psalm is rendered as , " The Lord is all that , " as opposed to what I knew it as , " The Lord is my shepherd . " Then I went down and I read several versions , and I wrote down every single law that I could find . And this was a very long list -- over 700 rules . And they range from the famous ones that I had heard of -- The Ten Commandments , love your neighbor , be fruitful and multiply . So I wanted to follow those . And actually I take my projects very seriously because I had twins during my year , so I definitely take my projects seriously . But I also wanted to follow the hundreds of arcane and obscure laws that are in the Bible . There is the law in Leviticus -- " You cannot shave the corners of your beard . " I 
 did n't know where my corners were , so I decided to let the whole thing grow , and this is what I looked like by the end . As you can imagine , I spent a lot of time at airport security . ( Laughter ) My wife would n't kiss me for the last two months . So , certainly the challenge was there . The Bible says you cannot wear clothes made of mixed fibers , so I thought , " Sounds strange , but I 'll try it . " You only know if you try it . I got rid of all my poly-cotton t-shirts . The Bible says that if two men are in a fight , and the wife of one of those men grabs the testicles of the other man , then her hand shall be cut off . So , I wanted to follow that rule . ( Laughter ) That one I followed by default , by not getting in a fight with a man whose wife was standing nearby , looking like she had a strong grip . ( Laughter ) So -- oh , there 's another shot of my beard . I will say it was an amazing year because it really was life-changing , and incredibly challenging . And there
  were two types of laws that were particularly challenging . The first was avoiding the little sins that we all commit every day . You know , I could spend a year not killing , but spending a year not gossiping , not coveting , not lying -- you know , I live in New York , and I work as a journalist , so this was 75 , 80 percent of my day I had to do it . But it was really interesting , because I was able to make some progress because I could n't believe how much my behavior changed my thoughts . This was one of the huge lessons of the year , is that I almost pretended to be a better person , and I became a little bit of a better person . So I had always thought , you know , " You change your mind , and you change your behavior , " but it 's often the other way round . You change your behavior , and you change your mind . So , you know , if you want to become more compassionate , you visit sick people in the hospital , and you will become more compassionate . You donate money to a ca
 use , and you become emotionally involved in that cause . So , it really was cognitive psychology -- you know , cognitive dissonance -- that I was experiencing . The Bible actually talks about cognitive psychology , very primitive cognitive psychology . In the Proverbs , it says that if you smile , you will become happier , which , as we know , is actually true . The second type of rule that was difficult to obey was the rules that will get you into a little trouble in 21st-century America . And perhaps the clearest example of this is stoning adulterers . ( Laughter ) But it 's a big part of the Bible , so I figured I had to address it . So , I was able to stone one adulterer . It happened -- I was in the park , and I was dressed in my biblical clothing -- sandals and a white robe -- you know , because again , the outer affects the inner . I wanted to see how dressing biblically affected my mind . And this man came up to me and he said , " Why are you dressed like that ? " And I exp
 lained my project , and he said , " Well , I am an adulterer , are you going to stone me ? " And I said , " Well , that would be great ! " ( Laughter ) And I took out a handful of stones from my pocket that I had been carrying around for weeks , hoping for just this interaction -- and , you know , they were pebbles -- but he grabbed them out of my hand . He was actually an elderly man , mid-seventies , just so you know . But he 's still an adulterer , and still quite angry . He grabbed them out of my hand and threw them at my face , and I felt that I could -- eye for an eye , I could retaliate , and throw one back at him . So that was my experience stoning , and it did allow me to talk about in a more serious way these big issues . How can the Bible be so barbaric in some places , and yet so incredibly wise in others ? How should we view the Bible ? Should we view it , you know , as original intent , like a sort of a Scalia version of the Bible ? How was the Bible written ? And actu
 ally , since this is a tech crowd , I talk in the book about how the Bible actually reminds me of Wikipedia because it has all of these authors and editors over hundreds of years . And it 's sort of evolved . It 's not a book that was written and came down from on high . So I thought I would end by telling you just a couple of the take-away , the bigger lessons that I learned from my year . The first is -- Thou shalt not take the Bible literally . This became very , very clear , early on . Because if you do , then you end up acting like a crazy person , and stoning adulterers , or -- here 's another example -- well , that 's another -- I did spend some time shepherding . ( Laughter ) It 's a very relaxing vocation . I recommend it . But this one is , the Bible says that you cannot touch women during certain times of the month , and more than that , you cannot sit on a seat where a menstruating woman has sat . And my wife thought this was very offensive , so she sat in every seat in 
 our apartment , and I had to spend much of the year standing until I bought my own seat and carried it around . So , you know , I met with creationists . I went to the creationists ' museum . And these are the ultimate literalists . And it was fascinating , because they were not stupid people at all . I would wager that their IQ is exactly the same as the average evolutionist . It 's just that their faith is so strong in this literal interpretation of the Bible that they distort all the data to fit their model . And they go through these amazing mental gymnastics to accomplish this . And I will say , though , the museum is gorgeous . They really did a fantastic job . If you 're ever in Kentucky , there 's -- you can see a movie of the flood , and they have sprinklers in the ceiling that will sprinkle on you during the flood scenes . So , whatever you think of creationism , and I think it 's crazy , they did a great job . ( Laughter ) Another lesson is that thou shalt give thanks . A
 nd this one was a big lesson because I was praying , giving these prayers of thanksgiving , which was odd for an agnostic . But I was saying thanks all the time , every day , and I started to change my perspective , and I started to realize the hundreds of little things that go right every day , that I did n't even notice , that I took for granted -- as opposed to focusing on the three or four that went wrong . So , this is actually a key to happiness for me , is to just remember when I came over here , the car did n't flip over , and I did n't trip coming up the stairs . It 's a remarkable thing . Third , that thou shall have reverence . This one was unexpected because I started the year as an agnostic , and by the end of the year I became what a friend of mine calls a reverent agnostic , which I love . And I 'm trying to start it as a movement . So if anyone wants to join , the basic idea is , whether or not there is a God , there 's something important and beautiful about the ide
 a of sacredness , and that our rituals can be sacred . The Sabbath can be sacred . This was one of the great things about my year , doing the Sabbath , because I am a workaholic , so having this one day where you cannot work -- it really , that changed my life . So , this idea of sacredness , whether or not there is a God . Thou shall not stereotype . This one happened because I spend a lot of time with various religious communities throughout America because I wanted it to be more than about my journey . I wanted it to be about religion in America . So I spent time with evangelical Christians , and Hasidic Jews and the Amish . I 'm very proud because I think I 'm the only person in America to out Bible-talk a Jehovah 's Witness . ( Laughter ) After three and a half hours , he looked at his watch , he 's like , " I gotta go . " ( Laughter ) Oh , thank you very much . Thank you . Bless you , bless you . But it was interesting because I had some very preconceived notions about , for i
 nstance , evangelical Christianity , and I found that it 's such a wide and varied movement that it is difficult to make generalizations about it . There 's a group I met with called the Red Letter Christians , and they focus on the red words in the Bible , which are the ones that Jesus spoke -- that 's how they printed them in the old bibles . And their argument is that Jesus never talked about homosexuality . They have a pamphlet that says , " Here 's what Jesus said about homosexuality , " and you open it up , and there 's nothing in it . So , they say Jesus did talk a lot about helping the outcasts , helping poor people . So this was very inspiring to me . I recommend Jim Wallace and Tony Campolo . They 're very inspiring leaders , even though I disagree with much of what they say . Also , thou shalt not disregard the irrational . This one was very unexpected because , you know , I grew up with the scientific worldview , and I was shocked learning how much of my life is governed
  by irrational forces . And the thing is , if they 're not harmful , they 're not to be completely dismissed . Because I learned that -- I was thinking , I was doing all these rituals , these biblical rituals , separating my wool and linen , and I would ask these religious people " Why would The Bible possibly tell us to do this ? Why would God care ? " And they said , " We do n't know , but it 's just rituals that give us meaning . " And I would say , " But that 's crazy . " And they would say , " Well , what about you ? You blow out candles on top of a birthday cake . If a guy from Mars came down and saw , here 's one guy blowing out the fire on top of a cake versus another guy not wearing clothes of mixed fabrics , would the Martians say , 'Well , that guy , he makes sense , but that guy 's crazy ? ' " So no , I think that rituals are , by nature , irrational . So the key is to choose the right rituals , the ones that are not harmful -- but rituals by themselves are not to be dis
 missed . And finally I learned that thou shall pick and choose . And this one I learned because I tried to follow everything in the Bible . And I failed miserably . Because you ca n't . You have to pick and choose , and anyone who follows the Bible is going to be picking and choosing . The key is to pick and choose the right parts . There 's the phrase called cafeteria religion , and the fundamentalists will use it in a denigrating way , and they 'll say , " Oh , it 's just cafeteria religion . You 're just picking and choosing . " But my argument is , " What 's wrong with cafeterias ? " I 've had some great meals at cafeterias . I 've also had some meals that make me want to dry heave . So , it 's about choosing the parts of the Bible about compassion , about tolerance , about loving your neighbor , as opposed to the parts about homosexuality is a sin , or intolerance , or violence , which are very much in the Bible as well . So if we are to find any meaning in this book , then we 
 have to really engage it , and wrestle with it . And I thought I 'd end with just a couple more . There 's me reading the Bible . That 's how I hailed taxi-cabs . ( Laughter ) Seriously , and it worked -- and yes , that was actually a rented sheep , so I had to return that in the morning , but it served well for a day . So , anyway , thank you so much for letting me speak . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/141ted_alan_kay_shares_a_powerful_idea_about_ideas.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/141ted_alan_kay_shares_a_powerful_idea_about_ideas.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/141ted_alan_kay_shares_a_powerful_idea_about_ideas.txt
new file mode 100644
index 0000000..bcc4f34
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/141ted_alan_kay_shares_a_powerful_idea_about_ideas.txt
@@ -0,0 +1,2 @@
+
+A great way to start , I think , with my view of simplicity , is to take a look at TED . Here you are , understanding why we 're here , what 's going on , with no difficulty at all . The best AI in the planet would find it complex and confusing , and my little dog Watson would find it simple and understandable , but would miss the point . ( Laughter ) He would have a great time . And of course , if you 're a speaker here , like Hans Rosling , a speaker finds this complex , tricky . But in Hans Rosling 's case , he had a secret weapon yesterday , literally , in his sword swallowing act . And I must say I thought of quite a few objects that I might try to swallow today and finally gave up on -- but he just did it and that was a wonderful thing . So Puck meant not only are we fools in the pejorative sense , but that we 're easily fooled . In fact what Shakespeare was pointing out is we go to the theater in order to be fooled , so we 're actually looking forward to it . We go to magic s
 hows in order to be fooled . And this makes many things fun , but it makes it difficult to actually get any kind of picture on the world we live in , or on ourselves . And our friend , Betty Edwards , the Drawing On the Right Side of the Brain lady , shows these two tables to her drawing class and says , the problem you have with learning to draw is not that you ca n't move your hand , but that the way your brain perceives images is faulty . It 's trying to perceive images into objects rather than seeing what 's there . And to prove it , she says , the exact size and shape of these tabletops is the same , and I 'm going to prove it to you . She does this with cardboard , but since I have an expensive computer here , I 'll just rotate this little guy around and ... . Now having seen that -- and I 've seen it hundreds of times , because I use this in every talk I give -- I still ca n't see that they 're the same size and shape , and I doubt that you can either . So what do artists do 
 ? Well , what artists do is to measure . They measure very , very carefully . And if you measure very , very carefully with a stiff arm and a straight edge , you 'll see that those two shapes are exactly the same size . And the Talmud saw this a long time ago , saying , we see things not as they are , but as we are . I certainly would like to know what happened to the person who had that insight back then , if they actually followed it to its ultimate conclusion . So if the world is not as it seems and we see things as we are , then what we call reality is a kind of hallucination happening inside here . It 's a waking dream . And understanding that that is what we actually exist in is one of the biggest epistemological barriers in human history . And what that means : " simple and understandable " might not be actually simple or understandable , and things we think are complex might be made simple and understandable . Somehow we have to understand ourselves to get around our flaws .
  We can think of ourselves as kind of a noisy channel . The way I think of it is , we ca n't learn to see until we admit we 're blind . Once you start down at this very humble level , then you can start finding ways to see things . And what 's happened over the last four hundred years in particular is that human beings have invented brainlets : little additional parts for our brain , made out of powerful ideas that help us see the world in different ways . And these are in the form of sensory apparatus -- telescopes , microscopes -- reasoning apparatus , various ways of thinking , and most importantly , in the ability to change perspective on things . I 'll talk about that a little bit . It 's this change in perspective , and what it is we think we 're perceiving , that has helped us make more progress in the last four hundred years than we have in the rest of human history . And yet it is not taught in any K through 12 curriculum in America that I 'm aware of . So one of the things
  that goes from simple to complex is when we do more . We like more . If we do more in a kind of a stupid way , the simplicity gets complex . And in fact , we can keep on doing it for a very long time . But Murray Gell-Mann yesterday talked about emergent properties . Another name for them could be " architecture " as a metaphor for taking the same old material and thinking about non-obvious , non-simple ways of combining it . And in fact , what Murray was talking about yesterday in the fractal beauty of nature , of having the descriptions at various levels be rather similar , all goes down to the idea that the elementary particles are both sticky and stand-offish , and they 're in violent motion . Those three things give rise to all the different levels of what seem to be complexity in our world . But how simple ? So when I saw the Roslings ' Gapminder stuff a few years ago , I just thought it was the greatest thing I 'd seen in conveying complex ideas simply . But then I had a tho
 ught of , boy , maybe it 's too simple . And I put some effort in to try and check to see how well these simple portrayals of trends over time actually matched up with some ideas and investigations from the side , and I found that they matched up very well . So the Roslings have been able to do simplicity without removing what 's important about the data . Whereas the film yesterday that we saw of the simulation of the inside of a cell , as a former molecular biologist , I did n't like that at all . Not because it was n't beautiful or anything , but because it misses the thing that most students fail to understand about molecular biology , and that is , why is there any probability at all of two complex shapes finding each other just the right way so they combine together and be catalyzed ? And what we saw yesterday was , every reaction was fortuitous . They just swooped in the air and bound , and something happened . But in fact those molecules are spinning at the rate of about a m
 illion revolutions per second . They 're agitating back and forth their size every two nanoseconds . They 're completely crowded together . They 're jammed , they 're bashing up against each other . And if you do n't understand that in your mental model of this stuff , what happens inside of a cell seems completely mysterious and fortuitous . And I think that 's exactly the wrong image for when you 're trying to teach science . So another thing that we do is to confuse adult sophistication with the actual understanding of some principle . So a kid who 's 14 in high school gets this version of the Pythagorean theorem , which is a truly subtle and interesting proof , but in fact it 's not a good way to start learning about mathematics . So a more direct one , one that gives you more of the feeling of math , is something closer to Pythagoras ' own proof which goes like this . So here we have this triangle , and if we surround that C square with three more triangles and we copy that , n
 otice that we can move those triangles down like this , and that leaves two open areas that are kind of suspicious , and bingo . And that is all you have to do . And this kind of proof is the kind of proof that you need to learn when you 're learning mathematics in order to get an idea of what it means before you look into the , literally , 12 or 1500 proofs of Pythagoras ' theorem that had been discovered . Now let 's go to young children . This is a very unusual teacher who was a kindergarten and first-grade teacher , but was a natural mathematician . So she was like that jazz musician friend you have who never studied music , but is a terrific musician . She just had a feeling for math , and here are her six-year-olds , and she 's got them making shapes out of a shape . So they pick a shape they like -- a diamond , or a square , or a triangle , or a trapezoid -- and then they try and make the next larger shape of that same shape , and the next larger shape . And you can see the t
 rapezoids are a little challenging there . And [ what ] this teacher did on every project was to have the children act like first it was a creative arts project and then something like science . So they 'd created these artifacts . Now she had them look at them and do this laborious -- which I thought for a long time , until she explained to me , was to slow them down so they 'll think . So they 're cutting out the little pieces of cardboard here , and pasting them up . But the whole point of this thing is for them to look at this chart and fill it out . What have you noticed about what you did ? And so six-year-old Lauren there noticed that the first one took one , and the second one took three more , and the total was four on that one . The third one took five more , and the total was nine on that one , and then the next one . So she saw right away that the additional tiles that you had to add around the edges was always going to grow by two . So she was very confident about how s
 he made those numbers there . And she could see that these were the square numbers up until about six . Where she was n't sure what six times six was , and what seven times seven was . But then she was confident again . So that 's what Lauren did . And then the teacher , Gillian Ishijima , had the kids bring all of their projects up to the front of the room and put them on the floor . And everybody went batshit . Holy shit ! They 're the same ! No matter what the shapes were , the growth law is the same . And the mathematicians and scientists in the crowd will recognize these two progressions as a first order discrete differential equation , and a second order discrete differential equation . Derived by six-year-olds . Well , that 's pretty amazing . That is n't what we usually try to teach six-year-olds . So let 's take a look now at how we might use the computer for some of this . And so the first idea here is just to show you the kind of things that children do . I 'm using the s
 oftware that we 're putting on the 100 dollar laptop . So I 'd like to draw a little car here . I 'll just do this very quickly . And put a big tire on him . And I get a little object here , and I can look inside this object . I 'll call it a car . And here 's a little behavior : car forward . Each time I click it , car turn . If I want to make a little script to do this over and over again , I just drag these guys out and set them going . And I can try steering the car here by -- see the car turn by five here ? So what if I click this down to zero ? It goes straight . That 's a bit of a revelation for nine-year-olds . Make it go in the other direction . But of course that 's a little bit like kissing your sister as far as driving a car . So the kids want to do a steering wheel . So they draw a steering wheel . And we 'll call this a wheel . And , see this wheel 's heading here ? If I turn this wheel , you can see that number over there going minus and positive . That 's kind of an 
 invitation to pick up this name of those numbers coming out there and to just drop it into the script here . And now I can steer the car with the steering wheel . And it 's interesting . You know how much trouble the children have with variables , but by learning it this way , in a situated fashion , they never forget from this single trial what a variable is and how to use it . And we can reflect here the way Gillian Ishijima did . So if you look at the little script here , the speed is always going to be 30. We 're going to move the car , according to that , over and over again . And I 'm dropping a little dot for each one of these things . They 're evenly spaced because they 're 30 apart . And what if I do this progression that the six-year-olds did of saying , OK , I 'm going to increase the speed by two each time , and then I 'm going to increase the distance by the speed each time ? What do I get there ? We get a visual pattern of what these nine-year-olds called acceleration 
 . So how did the children do science ? ( Video ) Teacher : Objects that you think will fall to the earth at the same time -- Kid : This is nice . Teacher : Do not pay any attention to what anybody else is doing . Who 's got the apple ? Alan Kay : They 've got little stopwatches . Teacher : What do you get ? What did you get ? AK : Stopwatches are n't accurate enough . Girl : 0.99 seconds . Teacher : So put " sponge ball " -- Girl : There was a shot-put and a sponge ball , because they 're two totally different weights . And if you drop them at the same time , maybe they 'll drop at the same speed . Teacher : Drop . AK : So obviously Aristotle never asked a child about this particular point , because of course he did n't bother doing the experiment , and neither did St. Thomas Aquinas . And it was not until Galileo actually did it that an adult thought like a child . Only 400 years ago . We get one child like that about every classroom of 30 kids who will actually cut straight to the
  chase . Now , what if we want to look at this more closely ? We can take a movie of what 's going on , but even if we single stepped this movie , it 's tricky to see what 's going on . And so what we can do is , we can lay out the frames side by side , or stack them up . So when the children see this , they say , " Ah , acceleration , " remembering back four months when they did their cars sideways , and they start measuring to find out what kind of acceleration it is . And so what I 'm doing is measuring from the bottom of one image to the bottom of the next image , about a fifth of a second later , like that , and they 're getting faster and faster each time . And if I stack these guys up , then we see the differences , the increase in the speed is constant . And they say , oh , yeah , constant acceleration . We 've done that already . And how shall we look and verify that we actually have it ? So we ca n't tell much from just making the ball drop there , but if we drop the ball 
 and run the movie at the same time , we can see that we have come up with an accurate physical model . Galileo , by the way , did this very cleverly by running a ball down backwards down the strings of his lute . I pulled out those apples to remind myself to tell you that this is actually probably a Newton and the apple-type story , but it 's a great story . And I thought I would do just one thing on the 100 dollar laptop here just to prove that this stuff works here . So once you have gravity , here 's this -- increase the speed by something , increase the ship 's speed . If I start the little game here that the kids have done , it 'll crash the space ship . But if I oppose gravity , here we go -- oops ! ( Laughter ) One more . Yeah , there we go . Yeah , OK ? I guess the best way to end this is with two quotes . Marshall McLuhan said , " Children are the messages that we send to the future . " But in fact , if you think of it , children are the future we send to the future . Forge
 t about messages . Children are the future . And children in the first and second world , and most especially in the third world , need mentors . And this summer we 're going to build 5 million of these 100 dollar laptops and maybe 50 million next year . But we could n't create a thousand new teachers this summer to save our life . And that means that we once again have a thing where we can put technology out , but the mentoring that is required to go from a simple new iChat instant messaging system to something with depth is missing . I believe this has to be done with a new kind of user interface . And this new kind of user interface could be done with an expenditure of about 100 million dollars . It sounds like a lot , but it is literally 18 minutes of what we 're spending in Iraq . We 're spending 8 billion dollars a month . 18 minutes is 100 million dollars . So this is actually cheap . And Einstein said , " Things should be as simple as possible , but not simpler . " Thank you
  . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/142ted_alex_tabarrok_foresees_economic_growth.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/142ted_alex_tabarrok_foresees_economic_growth.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/142ted_alex_tabarrok_foresees_economic_growth.txt
new file mode 100644
index 0000000..0b1394e
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/142ted_alex_tabarrok_foresees_economic_growth.txt
@@ -0,0 +1,2 @@
+
+The first half of the twentieth century was an absolute disaster in human affairs , a cataclysm . We had the First World War , the Great Depression , the Second World War , and the rise of the communist nations . And each one of these forces split the world , tore the world apart , divided the world . And they threw up walls , political walls , trade walls , transportation walls , communication walls , iron curtains , which divided peoples and nations . It was only in the second half of the twentieth century that we slowly began to pull ourselves out of this abyss . Trade walls began to come tumbling down . Here are some data on tariffs : starting at 40 percent , coming down to less than 5 percent . We globalized the world . And what does that mean ? It means that we extended cooperation across national boundaries . We made the world more cooperative . Transportation walls came tumbling down . You know in 1950 the typical ship carried 5,000 to 10,000 tons worth of goods . Today a co
 ntainer ship can carry 150,000 tons . It can be manned with a smaller crew , and unloaded faster than ever before . Communication walls , I do n't have to tell you , the internet , have come tumbling down . And of course the iron curtains , political walls have come tumbling down . Now all of this has been tremendous for the world . Trade has increased . Here is just a little bit of data . In 1990 exports from China to the United States -- 15 billion dollars . By 2007 , over 300 billion dollars . And perhaps most remarkably , at the beginning of the twenty-first century , really for the first time in modern history , growth extended to almost all parts of the world . So China , I 've already mentioned , beginning in 1978 , around the time of the death of Mao , growth -- ten percent a year . Year after year after year , absolutely incredible . Never before in human history have so many people been raised out of such great poverty , as happened in China . China is the world 's greates
 t anti-poverty program over the last three decades . India , starting a little bit later , but in 1990 , begetting tremendous growth . Incomes at that time less than 1,000 dollars per year . And over the next 18 years have almost tripled . Growth of six percent a year . Absolutely incredible . Now Africa , Sub-Saharan Africa , Sub-Saharan Africa has been the area of the world most resistant to growth . And we can see the tragedy of Africa in the first few bars here . Growth was negative . People were actually getting poorer than their parents . And sometimes even poorer than their grandparents had been . But at the end of the twentieth century , the beginning of the twenty-first century , we saw growth in Africa . And I think , as you 'll see , there 's reasons for optimism . Because I believe that the best is yet to come . Now why . On the cutting edge today it 's new ideas which are driving growth . And by that I mean it 's products for which the research and development costs are
  really high , and the manufacturing costs are low . More than ever before it is these types of ideas which are driving growth on the cutting edge . Now ideas have this amazing property . Thomas Jefferson , I think , really expressed this quite well . He said , " He who receives an idea from me receives instruction himself , without lessening mine . As he who lights his candle at mine receives light without darkening me . Or to put it slightly differently , one apple feeds one man , but an idea can feed the world . Now this is not new . This is practically not new to TEDsters . This is practically the model of TED . What is new is that the greater function of ideas is going to drive growth even more than ever before . This provides a reason why trade and globalization are even more important , more powerful than ever before , and are going to increase growth more than ever before . And to explain why this is so , I have a question . Suppose that there are two diseases . One of them 
 is rare , the other one is common . But if they are not treated they are equally severe . If you had to choose , which would you rather have ? The common disease or the rare disease ? Common . The common . I think that 's absolutely right . Why ? Because there are more drugs to treat common diseases than there are to treat rare diseases . The reason for this is incentives . It costs about the same to produce a new drug , whether that drug treats 1,000 people , 100,000 people , or a million people . But the revenues are much greater if the drug treats a million people . So the incentives are much larger to produce drugs which treat more people . To put this differently , larger markets save lives . In this case misery truly does love company . Now think about the following : if China and India were as rich as the United States is today , the market for cancer drugs would be eight times larger than it is now . Now we are not there yet , but it is happening . As other countries become 
 richer the demand for these pharmaceuticals is going to increase tremendously . And that means an increase incentive to do research and development , which benefits everyone in the world . Larger markets increase the incentive to produce all kinds of ideas . Whether it 's software , whether it 's a computer chip , whether it 's a new design . For the Hollywood people in the audience , it even explains why action movies have larger budgets than comedies . It 's because action movies translate easier into other languages and other cultures . So the market for those movies is larger . People are willing to invest more , and the budgets are larger . Alright . Well if larger markets increase the incentive to produce new ideas , how do we maximize that incentive ? It 's by having one world market , by globalizing the world . The way I like to put this is , one idea , ideas are meant to be shared , so one idea can serve one world , one market . One idea , one world , one market . Well how 
 else can we create new ideas ? That 's one reason . Globalize , trade . How else can we create new ideas ? Well , more idea creators . Now idea creators , they come from all walks of life . Artists and innovators , many of the people you 've seen on this stage . I 'm going to focus on scientists and engineers because I have some data on that , and I 'm a data person . Now , today , less than 1/10th of one percent of the world 's population are scientists and engineers . ( Laughter ) The United States has been an idea leader . A large fraction of those people are in the United States . But the U. S. is losing its idea leadership . And for that I am very grateful . That is a good thing . It is fortunate that we are becoming less of an idea leader because for too long the United States , and a handful of other developed countries , have shouldered the entire burden of research and development . But consider the following : if the world as a whole were as wealthy as the United States is
  now there would be more than five times as many scientists and engineers contributing to ideas which benefit everyone , which are shared by everyone . I think of the great Indian mathematician , Ramanujan . How many Ramanujans are there in India today toiling in the fields , barely able to feed themselves , when they could be feeding the world ? Now we 're not there yet . But it is going to happen in this century . The real tragedy of the last century is this : if you think about the world 's population as a giant computer , a massively parallel processor , then the great tragedy has been that billions of our processors have been off line . But in this century China is coming on line . India is coming on line . Africa is coming on line . We will see an Einstein in Africa in this century . Here is just some data . This is China . 1996 , less than one million new university students in China , per year . 2006 , over five million . Now think what this means . This means we all benefit
  when another country gets rich . We should not fear other countries becoming wealthy . That is something that we should embrace -- a wealthy China , a wealthy India , a wealthy Africa . We need a greater demand for ideas , those larger markets I was talking about earlier , and a greater supply of ideas for the world . Now you can see some of the reasons why I 'm optimistic . Globalization is increasing the demand for ideas , the incentive to create new ideas . Investments in education are increasing the supply of new ideas . In fact if you look at world history you can see some reasons for optimism . From about the beginnings of humanity to 1500 , zero economic growth , nothing . 1500 to 1800 , maybe a little bit of economic growth . But less in a century than you expect to see in a year today . 1900s maybe one percent . Twentieth century a little bit over two percent . Twenty-first century could easily be 3.3 even higher percent . Even at that rate by 2100 , average GDP per capita
  in the world will be 200,000 dollars . That 's not U. S. GDP per capita , which will be over a million . But world GDP per capita , 200,000 dollars . That 's not that far . We wo n't make it . But some of our grandchildren probably will . And I should say I think this is a rather modest prediction . In Kurzweilian terms this is gloomy . In Kurzweilian terms I 'm like the Eeyore of economic growth . ( Laughter ) Alright what about problems ? What about a great depression ? Well let 's take a look . Let 's take a look at the Great Depression . Here is GDP per capita from 1900 to 1929. Now let 's imagine that you were an economist in 1929 , trying to forecast future growth for the United States , not knowing that the economy was about to go off a cliff . Not knowing that we were about to enter the greatest economic disaster certainly in the twentieth century . What would you have predicted not knowing this ? If you had based your prediction , your forecast on 1900 to 1929 you 'd have 
 predicted something like this . If you 'd been a little more optimistic , say based upon the roaring 20s , you 'd have said this . So what actually happened ? We went off a cliff but we recovered . In fact in the second half of the twentieth century growth was even higher than anything you would have predicted based upon the first half of the twentieth century . So growth can wash away even what appears to be a great depression . Alright . What else ? Oil . Oil . This was a big topic . When I was writing up my notes oil was 140 dollars per barrel . So people were asking a question . The were saying , " Is China drinking our milkshake ? " ( Laughter ) And there is some truth to this in the sense that we have something of a finite resource . And increased growth is going to push up demand for that . But I think I do n't have to tell this audience that a higher price of oil is not necessarily a bad thing . Moreover , as everyone knows , look it 's energy , not oil , which counts . And 
 higher oil prices mean a greater incentive to invest in energy R&D . You can see this in the data . As oil prices go up , energy patents go up . The world is much better equipped to overcome an increase in the price of oil today , than ever in the past , because of what I 'm talking about . One idea , one world , one market . So I 'm optimistic so long as we hew to these two ideas : to keep globalizing world markets , keep extending cooperation across national boundaries , and keep investing in education . Now the United States has a particularly important role to play in this -- to keep our education system globalized , to keep our education system open to students from all over the world -- because our education system is the candle that other students come to to light their own candles . Now remember here what Jefferson said . Jefferson said , " When they come and light their candles at ours , that they gain light , and we are not darkened . " But Jefferson was n't quite right , 
 was he ? Because the truth is , when they light their candles at ours , there is twice as much light available for everyone . So my view is be optimistic . Spread the ideas . Spread the light . Thank you . ( Applause ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/143ted_amy_tan_on_creativity.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/143ted_amy_tan_on_creativity.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/143ted_amy_tan_on_creativity.txt
new file mode 100644
index 0000000..3f45de2
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/143ted_amy_tan_on_creativity.txt
@@ -0,0 +1,2 @@
+
+The value of nothing : out of nothing comes something . That was an essay I wrote when I was 11 years old and I got a B+ . ( Laughter ) What I 'm going to talk about : nothing out of something , and how we create . And I 'm gonna try and do that within the 18-minute time span that we were told to stay within , and to follow the TED commandments : that is , actually , something that creates a near-death experience , but near-death is good for creativity . ( Laughter ) OK . So , I also want to explain , because Dave Eggers said he was going to heckle me if I said anything that was a lie , or not true to universal creativity . And I 've done it this way for half the audience , who is scientific . When I say we , I do n't mean you , necessarily ; I mean me , and my right brain , my left brain , and the one that 's in between that is the censor and tells me what I 'm saying is wrong . And I 'm going do that also by looking at what I think is part of my creative process , which includes a
  number of things that happened , actually -- the nothing started even earlier than the moment in which I 'm creating something new . And that includes nature , and nurture , and what I refer to as nightmares . Now in the nature area , we look at whether or not we are innately equipped with something , perhaps in our brains , some abnormal chromosome that causes this muse-like effect . And some people would say that we 're born with it in some other means , and others , like my mother , would say that I get my material from past lives . Some people would also say that creativity may be a function of some other neurological quirk -- van Gogh syndrome -- that you have a little bit of , you know , psychosis , or depression . I do have to say , somebody -- I read recently that van Gogh was n't really necessarily psychotic , that he might have had temporal lobe seizures , and that might have caused his spurt of creativity , and I do n't -- I suppose it does something in some part of your
  brain . And I will mention that I actually developed temporal lobe seizures a number of years ago , but it was during the time I was writing my last book , and some people say that book is quite different . I think that part of it also begins with a sense of identity crisis : you know , who am I , why am I this particular person , why am I not black like everybody else ? And sometimes you 're equipped with skills , but they may not be the kind of skills that enable creativity . I used to draw . I thought I would be an artist . And I had a miniature poodle . And it was n't bad , but it was n't really creative . Because all I could really do was represent in a very one-on-one way . And I have a sense that I probably copied this from a book . And then I also was n't really shining in a certain area that I wanted to be , and you know , you look at those scores , and it was n't bad , but it was not certainly predictive that I would one day make my living out of the artful arrangement of
  words . Also , one of the principles of creativity is to have a little childhood trauma . And I had the usual kind that I think a lot of people had , and that is that , you know , I had expectations placed on me . That figure right there , by the way , figure right there was a toy given to me when I was but nine years old , and it was to help me become a doctor from a very early age . I have some ones that were long lasting : from the age of five to 15 , this was supposed to be my side occupation , and it led to a sense of failure . But actually there was something quite real in my life that happened when I was about 14. And it was discovered that my brother , in 1967 , and then my father , six months later , had brain tumors . And my mother believed that something had gone wrong , and she was gonna find out what it was . And she was gonna fix it . My father was a Baptist minister , and he believed in miracles , and that God 's will would take care of that . But of course , they en
 ded up dying , six months apart . And after that , my mother believed that it was fate , or curses -- she went looking through all the reasons in the universe why this would have happened . Everything except randomness . She did not believe in randomness . There was a reason for everything . And one of the reasons , she thought , was that her mother , who had died when she was very young , was angry at her . And so I had this notion of death all around me because my mother also believed that I would be next , and she would be next And when you are faced with the prospect of death very soon , you begin to think very much about everything . You become very creative , in a survival sense . And this , then , led to my big questions . And they 're the same ones that I have today . And they are : Why do things happen , and how do things happen ? And the one my mother asked : How do I make things happen ? It 's a wonderful way to look at these questions , when you write a story . Because a
 fter all , in that framework , between page one and 300 , you have to answer this question of why things happen , how things happen , in what order they happen . What are the influences ? How do I , as the narrator , as the writer , also influence that ? And it 's also one that I think many of our scientists have been asking . It 's a kind of cosmology , and I have to develop a cosmology of my own universe , as the creator of that universe . And you see , there 's a lot of back and forth in trying to make that happen , trying to figure it out -- years and years , oftentimes . So when I look at creativity , I also think that it is this sense or this inability to repress my looking at associations in practically anything in life . And I got a lot of them during what 's been going on throughout this conference , almost everything that 's been going on . And so I 'm going to use , as the metaphor , this association : quantum mechanics , which I really do n't understand , but I 'm still 
 gonna use it as the process for explaining how it is the metaphor . So in quantum mechanics , of course , you have dark energy and dark matter . And it 's the same thing in looking at these questions of how things happen . There 's a lot of unknown , and you often do n't know what it is except by its absence . But when you make those associations , you want them to come together in a kind of synergy in the story , and what you 're finding is what matters . The meaning . And that 's what I look for in my work , a personal meaning . There is also the uncertainty principle , which is part of quantum mechanics , as I understand it . ( Laughter ) And this happens constantly in the writing . And there 's the terrible and dreaded observer effect , in which you 're looking for something , and you know , things are happening simultaneously , and you 're looking at it in a different way , and you 're trying to really look for the about-ness . Or what is this story about . And if you try too h
 ard , then you will only write the about . You wo n't discover anything . And what you were supposed to find , what you hoped to find , in some serendipitous way , is no longer there . Now , I do n't want to ignore the other side of what happens in our universe , like many of our scientists have . And so I am going to just throw in string theory here , and just say that creative people are multi-dimensional , and there are eleven levels , I think , of anxiety . ( Laughter ) And they all operate at the same time . There is also a big question of ambiguity . And I would link that to something called the cosmological constant . And you do n't know what is operating , but something is operating there . And ambiguity , to me , is very uncomfortable in my life , and I have it . Moral ambiguity . It is constantly there . And just as an example , this is one that recently came to me . It was something I read in an editorial by a woman who was talking about the war in Iraq . And she said , "
  Save a man from drowning , you are responsible to him for life . " A very famous Chinese saying , she said . And that means because we went into Iraq , we should stay there until things were solved . You know , maybe even 100 years . So there was another one that I came across , and it 's " saving fish from drowning . " And it 's what Buddhist fishermen say , because they 're not supposed to kill anything . And they also have to make a living , and people need to be fed . So their way of rationalizing that is they are saving the fish from drowning , and unfortunately in the process the fish die . Now what 's encapsulated in both these drowning metaphors -- actually , one of them is my mother 's interpretation , and it is a famous Chinese saying because she said it to me : " Save a man from drowning , you are responsible to him for life . " And it was a warning -- do n't get involved in other people 's business , or you 're going to get stuck . OK . I think if somebody really was dr
 owning , she 'd save them . But both of these sayings , saving a fish from drowning , or saving a man from drowning , to me they had to do with intentions . And all of us in life , when we see a situation , we have a response . And then we have intentions . There 's an ambiguity of what that should be that we should do , and then we do something . And the results of that may not match what our intentions had been . Maybe things go wrong . And so , after that , what are our responsibilities ? What are we supposed to do ? Do we stay in for life , or do we do something else and justify and say , well , my intentions were good , and therefore I cannot be held responsible for all of it ? That is the ambiguity in my life that really disturbed me , and led me to write a book called Saving Fish From Drowning . I saw examples of that , once I identified this question . It was all over the place . I got these hints everywhere . And then , in a way , I knew that they had always been there . An
 d then writing , that 's what happens . I get these hints , these clues , and I realize that they 've been obvious , and yet they have not been . And what I need , in effect , is a focus . And when I have the question , it is a focus . And all these things that seem to be flotsam and jetsam in life actually go through that question , and what happens is those particular things become relevant . And it seems like it 's happening all the time . You think there 's a sort of coincidence going on , a serendipity , in which you 're getting all this help from the universe . And it may also be explained that now you have a focus . And you are noticing it more often . But you apply this . You begin to look at things having to do with your tensions . Your brother , who 's fallen in trouble , do you take care of him ? Why or why not ? It may be something that is perhaps more serious -- as I said , human rights in Burma . I was thinking that I should n't go because somebody said if I did , it w
 ould show that I approved of the military regime there . And then after a while , I had to ask myself , " Why do we take on knowledge , why do we take on assumptions that other people have given us ? " And it was the same thing that I felt when I was growing up , and was hearing these rules of moral conduct from my father , who was a Baptist minister . So I decided that I would go to Burma for my own intentions , and still did n't know that if I went there , what the result of that would be if I wrote a book -- and I just would have to face that later , when the time came . We are all concerned with things that we see in the world that we are aware of . We come to this point and say , what do I as an individual do ? Not all of us can go to Africa , or work at hospitals , so what do we do if we have this moral response , this feeling ? Also , I think one of the biggest things we are all looking at , and we talked about today , is genocide . This leads to this question , when I look a
 t all these things that are morally ambiguous and uncomfortable , and I consider what my intentions should be , I realize it goes back to this identity question that I had when I was a child -- and why am I here , and what is the meaning of my life , and what is my place in the universe ? It seems so obvious , and yet it is not . We all hate moral ambiguity in some sense , and yet it is also absolutely necessary . In writing a story , it is the place where I begin . Sometimes I get help from the universe , it seems . My mother would say it was the ghost of my grandmother from the very first book , because it seemed I knew things I was not supposed to know . Instead of writing that the grandmother died accidentally , from an overdose of opium while having too much of a good time , I actually put down in the story that the woman killed herself , and that actually was the way it happened . And my mother decided that that information must have come from my grandmother . There are also t
 hings , quite uncanny , which bring me information that will help me in the writing of the book . In this case , I was writing a story that included some kind of detail , period of history , a certain location . And I needed to find something historically that would match that . And I took down this book , and I -- first page that I flipped it to was exactly the setting , and the time period . And the kind of character I needed was the Taiping rebellion , happening in the area near Guilin , outside of that , and a character who thought he was the son of God . You wonder , are these things random chance ? Well , what is random ? What is chance ? What is luck ? What are things that you get from the universe that you ca n't really explain ? And that goes into the story too . These are the things I constantly think about from day to day . Especially when good things happen , and in particular , when bad things happen . But I do think there 's a kind of serendipity , and I do want to kno
 w what those elements are , so I can thank them , and also try to find them in my life . Because , again , I think that when I am aware of them , more of them happen . Another chance encounter is when I went to a place -- I just was with some friends , and we drove randomly to a different place , and we ended up in this non-tourist location , a beautiful village , pristine . And we walked three valleys beyond , and the third valley , there was something quite mysterious and ominous , a discomfort I felt . And then I knew that had to be setting of my book . And in writing one of the scenes , it happened in that third valley . For some reason I wrote about cairns -- stacks of rocks -- that a man was building . And I did n't know exactly why I had it , but it was so vivid . I got stuck , and a friend , when she asked if I would go for a walk with her dogs , that I said , sure . And about 45 minutes later , walking along the beach , I came across this . And it was a man , a Chinese man 
 , and he was stacking these things , not with glue , not with anything . And I asked him how is it possible to do this ? And he said , well , I guess with everything in life , there 's a place of balance . And this was exactly the meaning of my story at that point . I had so many examples -- I have so many instances like this when I 'm writing a story , and I cannot explain it . Is it because I had the filter that I have such a strong coincidence in writing about these things ? Or is it a kind of serendipity that we cannot explain , like the cosmological constant ? A big thing that I also think about is accidents . And as I said , my mother did not believe in randomness . What is the nature of accidents ? And how are we going to assign what the responsibility and the causes are , outside of a court of law ? I was able to see that in a firsthand way , when I went to beautiful Dong village , in Guizhou , the poorest province of China . And I saw this beautiful place . I knew I wanted 
 to come back . And I had a chance to do that when National Geographic asked me if I wanted to write anything about China . And I said yes , about this village of Singing people , Singing minority . And they agreed , and between the time I saw this place and the next time I went , there was a terrible accident . A man , an old man , fell asleep , and his quilt dropped in a pan of fire that kept him warm . 60 homes were destroyed , and 40 were damaged . Responsibility was assigned to the family . The man 's sons were banished to live three kilometers away , in a cow shed . And of course , as Westerners , we say , " Well , it was an accident . That 's not fair . It 's the son , not the father . " And when I go on a story , I have to let go of those kinds of beliefs . It takes a while , but I have to let go of them and just go there , and be there . And so I was there on three occasions , different seasons . And I began to sense something different about the history and what had happene
 d before , and the nature of life in a very poor village , and what you find as your joys , and your rituals , your traditions , your links with other families . And I saw how this had a kind of justice in its responsibility . I was able to find out also about the ceremony that they were using , a ceremony they had n't used in about 29 years . And it was to send some men -- a Feng Shui master sent men down to the underworld on ghost horses . Now you , as Westerners , and I , as Westerners , would say well , that 's superstition . But after being there for a while , and seeing the amazing things that happened , you begin to wonder whose beliefs are those that are in operation in the world , determining how things happen . So I remained with them , and the more I wrote that story , the more I got in to those beliefs , and I think that 's important for me -- to take on the beliefs , because that is where the story is real , and that is where I 'm gonna find the answers to how I feel ab
 out certain questions that I have in life . Years go by , of course , and the writing , it does n't happen instantly , as I 'm trying to convey it to you here at TED . The book comes and it goes . When it arrives , it is no longer my book . It is in the hands of readers , and they interpret it differently . But I go back to this question of , how do I create something out of nothing ? And how do I create my own life ? And I think it is by questioning , and saying to myself that there are no absolute truths . I believe in specifics , the specifics of story , and the past , the specifics of that past , and what is happening in the story at that point . I also believe that in thinking about things , my thinking about luck , and fate , and coincidences and accidents , God 's will , and the synchrony of mysterious forces , I will come to some notion of what that is , how we create . I have to think of my role . Where I am in the universe , and did somebody intend for me to be that way , 
 or is it just something I came up with ? And I also can find that by imagining fully , and becoming what is imagined , and yet is in that real world , the fictional world . And that is how I find particles of truth , not the absolute truth , or the whole truth . And they have to be in all possibilities , including those I never considered before . So there are never complete answers . Or rather , if there is an answer , it is to remind myself that there is uncertainty in everything , and that is good . Because then I will discover something new . And if there is a partial answer , a more complete answer from me , it is to simply imagine . And to imagine is to put myself in that story , until there was only -- there is a transparency between me and the story I am creating . And that 's how I 've discovered that if I feel what is in the story -- in one story -- then I come the closest , I think , to knowing what compassion is , to feeling that compassion . Because for everything , in 
 that question of how things happen , it has to do with the feeling . I have to become the story in order to understand a lot of that . We 've come to the end of the talk , and I will reveal what is in the bag , and it is the muse , and it is the things that transform in our lives , that are wonderful and stay with us . There she is . Thank you very much ! ( Applause ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/144ted_barry_schwartz_on_the_paradox_of_choice.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/144ted_barry_schwartz_on_the_paradox_of_choice.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/144ted_barry_schwartz_on_the_paradox_of_choice.txt
new file mode 100644
index 0000000..4310a9c
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/144ted_barry_schwartz_on_the_paradox_of_choice.txt
@@ -0,0 +1,2 @@
+
+I 'm going to talk to you about some stuff that 's in this book of mine that I hope will resonate with other things you 've already heard , and I 'll try to make some connections myself , in case you miss them . I want to start with what I call the " official dogma . " The official dogma of what ? The official dogma of all western industrial societies . And the official dogma runs like this : if we are interested in maximizing the welfare of our citizens , the way to do that is to maximize individual freedom . The reason for this is both that freedom is in and of itself good , valuable , worthwhile , essential to being human . And because if people have freedom , then each of us can act on our own to do the things that will maximize our welfare , and no one has to decide on our behalf . The way to maximize freedom is to maximize choice . The more choice people have , the more freedom they have , and the more freedom they have , the more welfare they have . This , I think , is so dee
 ply embedded in the water supply that it would n't occur to anyone to question it . And it 's also deeply embedded in our lives . I 'll give you some examples of what modern progress has made possible for us . This is my supermarket . Not such a big one . I want to say just a word about salad dressing . 175 salad dressings in my supermarket , if you do n't count the 10 different extra-virgin olive oils and 12 balsamic vinegars you could buy to make a very large number of your own salad dressings , in the off chance that none of the 175 the store has on offer suit you . So this is what the supermarket is like . And then you go to the consumer electronics store to set up a stereo system -- speakers , CD player , tape player , tuner , amplifier . And in this one single consumer electronics store , there are that many stereo systems . We can construct six and a half million different stereo systems out of the components that are on offer in one store . You 've got to admit that 's a lot
  of choice . In other domains -- the world of communications . There was a time , when I was a boy , when you could get any kind of telephone service you wanted , as long as it came from Ma Bell . You rented your phone . You did n't buy it . One consequence of that , by the way , is that the phone never broke . And those days are gone . We now have an almost unlimited variety of phones , especially in the world of cell phones . These are cell phones of the future . My favorite is the middle one -- the MP3 player , nose hair trimmer , and creme brulee torch . And if by some chance you have n't seen that in your store yet , you can rest assured that one day soon you will . And what this does is it leads people to walk into their stores asking this question . And do you know what the answer to this question now is ? The answer is " No. " It is not possible to buy a cell phone that does n't do too much . So , in other aspects of life that are much more significant than buying things , T
 he same explosion of choice is true . Health care -- it is no longer the case in the United States that you go to the doctor , and the doctor tells you what to do . Instead , you go to the doctor , and the doctor tells you , well , we could do A , or we could do B. A has these benefits , and these risks . B has these benefits , and these risks . What do you want to do ? And you say , " Doc , what should I do ? " And the doc says , A has these benefits and risks , and B has these benefits and risks . What do you want to do ? And you say , " If you were me , Doc , what would you do ? " And the doc says , " But I 'm not you . " And the result is -- we call it " patient autonomy , " which makes it sound like a good thing . But what it really is is a shifting of the burden and the responsibility for decision-making from somebody who knows something -- namely the doctor -- to somebody who knows nothing and is almost certainly sick and thus not in the best shape to be making decisions -- n
 amely the patient . There 's enormous marketing of prescription drugs to people like you and me , which , if you think about it , makes no sense at all , since we ca n't buy them . Why do they market to us if we ca n't buy them ? The answer is that they expect us to call our doctors the next morning and ask prescriptions to be changed . Something as dramatic as our identity has now become a matter of choice , as this slide is meant to indicate . We do n't inherit an identity , we get to invent it . And we get to re-invent ourselves as often as we like . And that means that every day when you wake up in the morning , you have to decide what kind of person you want to be . With respect to marriage and family , there was a time when the default assumption that almost everyone had is that you got married as soon as you could , and then you started having kids as soon as you could . The only real choice was who , not when , and not what you did after . Nowadays , everything is very much 
 up for grabs . I teach wonderfully intelligent students , and I assign 20 percent less work than I used to . And it 's not because they 're less smart , and it 's not because they 're less diligent . It 's because they are preoccupied , asking themselves , " Should I get married or not ? Should I get married now ? Should I get married later ? Should I have kids first , or a career first ? " All of these are consuming questions . And they 're going to answer these questions , whether or not it means not doing all the work I assign and not getting a good grade in my courses . And indeed they should . These are important questions to answer . Work -- we are blessed , as Carl was pointing out , with the technology that enables us to work every minute of every day from any place on the planet -- except the Randolph Hotel . ( Laughter ) There is one corner , by the way , that I 'm not going to tell anybody about , where the WiFi works . I 'm not telling you about it because I want to use 
 it . So what this means , this incredible freedom of choice we have with respect to work , is that we have to make a decision , again and again and again , about whether we should or should n't be working . We can go to watch our kid play soccer , and we have our cell phone on one hip , and our Blackberry on our other hip , and our laptop , presumably , on our laps . And even if they 're all shut off , every minute that we 're watching our kid mutilate a soccer game , we are also asking ourselves , " Should I answer this cell phone call ? Should I respond to this email ? Should I draft this letter ? " And even if the answer to the question is " no , " it 's certainly going to make the experience of your kid 's soccer game very different than it would 've been . So everywhere we look , big things and small things , material things and lifestyle things , life is a matter of choice . And the world we used to live in looked like this . That is to say , there were some choices , but not 
 everything was a matter of choice . And the world we now live in looks like this . And the question is , is this good news , or bad news ? And the answer is yes . ( Laughter ) We all know what 's good about it , so I 'm going to talk about what 's bad about it . All of this choice has two effects , two negative effects on people . One effect , paradoxically , is that it produces paralysis , rather than liberation . With so many options to choose from , people find it very difficult to choose at all . I 'll give you one very dramatic example of this , a study that was done of investments in voluntary retirement plans . A colleague of mine got access to investment records from Vanguard , the gigantic mutual fund company of about a million employees and about 2,000 different workplaces . And what she found is that for every 10 mutual funds the employer offered , rate of participation went down two percent . You offer 50 funds -- 10 percent fewer employees participate than if you only o
 ffer five . Why ? Because with 50 funds to choose from , it 's so damn hard to decide which fund to choose that you 'll just put it off until tomorrow . And then tomorrow , and then tomorrow , and tomorrow , and tomorrow , and of course tomorrow never comes . Understand that not only does this mean that people are going to have to eat dog food when they retire because they do n't have enough money to put away , it also means that making the decision is so hard that they pass up significant matching money from the employer . By not participating , they are passing up as much as 5,000 dollars a year from the employer , who would happily match their contribution . So paralysis is a consequence of having too many choices . And I think it makes the world look like this . ( Laughter ) You really want to get the decision right if it 's for all eternity , right ? You do n't want to pick the wrong mutual fund , or even the wrong salad dressing . So that 's one effect . The second effect is t
 hat even if we manage to overcome the paralysis and make a choice , we end up less satisfied with the result of the choice than we would be if we had fewer options to choose from . And there are several reasons for this . One of them is that with a lot of different salad dressings to choose from , if you buy one , and it 's not perfect -- and , you know , what salad dressing is ? It 's easy to imagine that you could have made a different choice that would have been better . And what happens is this imagined alternative induces you to regret the decision you made , and this regret subtracts from the satisfaction you get out of the decision you made , even if it was a good decision . The more options there are , the easier it is to regret anything at all that is disappointing about the option that you chose . Second , what economists call opportunity costs . Dan Gilbert made a big point this morning of talking about how much the way in which we value things depends on what we compare 
 them to . Well , when there are lots of alternatives to consider , it is easy to imagine the attractive features of alternatives that you reject , that make you less satisfied with the alternative that you 've chosen . Here 's an example . For those of you who are n't New Yorkers , I apologize . ( Laughter ) But here 's what you 're supposed to be thinking . Here 's this couple on the Hamptons . Very expensive real estate . Gorgeous beach . Beautiful day . They have it all to themselves . What could be better ? " Well , damn it , " this guy is thinking , " It 's August . Everybody in my Manhattan neighborhood is away . I could be parking right in front of my building . " And he spends two weeks nagged by the idea that he is missing the opportunity , day after day , to have a great parking space . Opportunity costs subtract from the satisfaction we get out of what we choose , even when what we choose is terrific . And the more options there are to consider , the more attractive featu
 res of these options are going to be reflected by us as opportunity costs . Here 's another example . Now this cartoon makes a lot of points . It makes points about living in the moment as well , and probably about doing things slowly . But one point it makes is that whenever you 're choosing one thing , you 're choosing not to do other things . And those other things may have lots of attractive features , and it 's going to make what you 're doing less attractive . Third : escalation of expectations . This hit me when I went to replace my jeans . I wear jeans almost all the time . And there was a time when jeans came in one flavor , and you bought them , and they fit like crap , and they were incredibly uncomfortable , and if you wore them long enough and washed them enough times , they started to feel OK . So I went to replace my jeans after years and years of wearing these old ones , and I said , " You know , I want a pair of jeans , here 's my size . " And the shopkeeper said , 
 " Do you want slim fit , easy fit , relaxed fit ? You want button fly or zipper fly ? You want stonewashed or acid washed ? Do you want them distressed ? You want boot cut , you want tapered , blah blah blah ... " On and on he went . My jaw dropped , and after I recovered , I said , " I want the kind that used to be the only kind . " ( Laughter ) He had no idea what that was , so I spent an hour trying on all these damn jeans , and I walked out of the store -- truth be told -- with the best fitting jeans I had ever had . I did better . All this choice made it possible for me to do better . But I felt worse . Why ? I wrote a whole book to try and explain this to myself . The reason I felt worse is that , with all of these options available , my expectations about how good a pair of jeans should be went up . I had very low expectations . I had no particular expectations when they only came in one flavor . When they came in 100 flavors , damn it , one of them should 've been perfect . 
 And what I got was good , but it was n't perfect . And so I compared what I got to what I expected , and what I got was disappointing in comparison to what I expected . Adding options to people 's lives ca n't help but increase the expectations people have about how good those options will be . And what that 's going to produce is less satisfaction with results , even when they 're good results . Nobody in the world of marketing knows this . Because if they did , you would n't all know what this was about . The truth is more like this . ( Laughter ) The reason that everything was better back when everything was worse is that when everything was worse , it was actually possible for people to have experiences that were a pleasant surprise . Nowadays , the world we live in -- we affluent , industrialized citizens , with perfection the expectation -- the best you can ever hope for is that stuff is as good as you expect it to be . You will never be pleasantly surprised because your expec
 tations , my expectations , have gone through the roof . The secret to happiness -- this is what you all came for -- the secret to happiness is low expectations . ( Laughter ) ( Applause ) I want to say -- just a little autobiographical moment -- that I actually am married to a wife , and she 's really quite wonderful . I could n't have done better . I did n't settle . But settling is n't always such a bad thing . Finally , one consequence of buying a bad-fitting pair of jeans when there is only one kind to buy is that when you are dissatisfied , and you ask why , who 's responsible , the answer is clear . The world is responsible . What could you do ? When there are hundreds of different styles of jeans available , and you buy one that is disappointing , and you ask why , who 's responsible ? It is equally clear that the answer to the question is you . You could have done better . With a hundred different kinds of jeans on display , there is no excuse for failure . And so when peop
 le make decisions , and even though the results of the decisions are good , they feel disappointed about them , they blame themselves . Clinical depression has exploded in the industrial world in the last generation . I believe a significant -- not the only , but a significant contributor to this explosion of depression , and also suicide , is that people have experiences that are disappointing because their standards are so high . And then when they have to explain these experiences to themselves , they think they 're at fault . And so the net result is that we do better in general , objectively , and we feel worse . So let me remind you . This is the official dogma , the one that we all take to be true , and it 's all false . It is not true . There 's no question that some choice is better than none , but it does n't follow from that that more choice is better than some choice . There 's some magical amount . I do n't know what it is . I 'm pretty confident that we have long since
  passed the point where options improve our welfare . Now , as a policy matter -- I 'm almost done -- as a policy matter , the thing to think about is this . What enables all of this choice in industrial societies is material affluence . There are lots of places in the world , and we have heard about several of them , where their problem is not that they have too much choice . Their problem is that they have too little . So the stuff I 'm talking about is the peculiar problem of modern , affluent , Western societies . And what is so frustrating and infuriating is this : Steve Levitt talked to you yesterday about how these expensive and difficult to install child seats do n't help . It 's a waste of money . What I 'm telling you is that these expensive , complicated choices -- it 's not simply that they do n't help . They actually hurt . They actually make us worse off . If some of what enables people in our societies to make all of the choices we make were shifted to societies in wh
 ich people have too few options , not only would those people 's lives be improved , but ours would be improved also . This is what economists call a Pareto-improving move . Income redistribution will make everyone better off -- not just poor people -- because of how all this excess choice plagues us . So to conclude . You 're supposed to read this cartoon , and , being a sophisticated person , say , " Ah ! What does this fish know ? You know nothing is possible in this fishbowl . " Impoverished imagination , a myopic view of the world -- and that 's the way I read it at first . The more I thought about it , however , the more I came to the view that this fish knows something . Because the truth of the matter is that if you shatter the fishbowl so that everything is possible , you do n't have freedom . You have paralysis . If you shatter this fishbowl so that everything is possible , you decrease satisfaction . You increase paralysis , and you decrease satisfaction . Everybody needs
  a fishbowl . This one is almost certainly too limited -- perhaps even for the fish , certainly for us . But the absence of some metaphorical fishbowl is a recipe for misery , and , I suspect , disaster . Thank you very much . ( Applause ) 
\ No newline at end of file


[32/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/61FictGrimmJ_HanselGretel_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/61FictGrimmJ_HanselGretel_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/61FictGrimmJ_HanselGretel_EN.txt.txt
new file mode 100644
index 0000000..3c63ab1
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/61FictGrimmJ_HanselGretel_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Hansel and Grethel Near a great forest there lived a poor woodcutter and his wife , and his two children ; the boy 's name was Hansel and the girl 's Grethel . They had very little to bite or to sup , and once , when there was great dearth in the land , the man could not even gain the daily bread . As he lay in bed one night thinking of this , and turning and tossing , he sighed heavily , and said to his wife , \u201c What will become of us ? we cannot even feed our children ; there is nothing left for ourselves . \u201d \u201c I will tell you what , husband , \u201d answered the wife ; \u201c we will take the children early in the morning into the forest , where it is thickest ; we will make them a fire , and we will give each of them a piece of bread , then we will go to our work and leave them alone ; they will never find the way home again , and we shall be quit of them . \u201d \u201c No , wife , \u201d said the man , \u201c I cannot do that ; I cannot find in my heart to take my children into the fo
 rest and to leave them there alone ; the wild animals would soon come and devour them . \u201d \u201c O you fool , \u201d said she , \u201c then we will all four starve ; you had better get the coffins ready , \u201d and she left him no peace until he consented . \u201c But I really pity the poor children , \u201d said the man . The two children had not been able to sleep for hunger , and had heard what their step-mother had said to their father . Grethel wept bitterly , and said to Hansel , \u201c It is all over with us . \u201d \u201c Do be quiet , Grethel , \u201d said Hansel , \u201c and do not fret ; 1 will manage something . \u201d And when the parents had gone to sleep he got up , put on his little coat , opened the back door , and slipped out . The moon was shining brightly , and the white flints that lay in front of the house glistened like pieces of silver . Hansel stooped and filled the little pocket of his coat as full as it would hold . Then he went back again , and said to Grethel , \u201c Be easy , dear littl
 e sister , and go to sleep quietly ; God will not forsake us , \u201d and laid himself down again in his bed . When the day was breaking , and before the sun had risen , the wife came and awakened the two children , saying , \u201c Get up , you lazy bones ; we are going into the forest to cut wood . \u201d Then she gave each of them a piece of bread , and said , \u201c That is for dinner , and you must not eat it before then , for you will get no more . \u201d Grethel carried the bread under her apron , for Hansel had his pockets full of the flints . Then they set off all together on their way to the forest . When they had gone a little way Hansel stood still and looked back towards the house , and this he did again and again , till his father said to him , \u201c Hansel , what are you looking at ? take care not to forget your legs . \u201d \u201c O father , \u201d said Hansel , \u201c lam looking at my little white kitten , who is sitting up on the roof to bid me good-bye . \u201d \u201c You young fool , \u201d said the
  woman , \u201c that is not your kitten , but the sunshine on the chimney-pot . \u201d Of course Hansel had not been looking at his kitten , but had been taking every now and then a flint from his pocket and dropping it on the road . When they reached the middle of the forest the father told the children to collect wood to make a fire to keep them , warm ; and Hansel and Grethel gathered brushwood enough for a little mountain j and it was set on fire , and when the flame was burning quite high the wife said , \u201c Now lie down by the fire and rest yourselves , you children , and we will go and cut wood ; and when we are ready we will come and fetch you . \u201d So Hansel and Grethel sat by the fire , and at noon they each ate their pieces of bread . They thought their father was in the wood all the time , as they seemed to hear the strokes of the axe : but really it was only a dry branch hanging to a withered tree that the wind moved to and fro . So when they had stayed there a long time thei
 r eyelids closed with weariness , and they fell fast asleep . When at last they woke it was night , and Grethel began to cry , and said , \u201c How shall we ever get out of this wood ? \u201c But Hansel comforted her , saying , \u201c Wait a little while longer , until the moon rises , and then we can easily find the way home . \u201d And when the full moon got up Hansel took his little sister by the hand , and followed the way where the flint stones shone like silver , and showed them the road . They walked on the whole night through , and at the break of day they came to their father 's house . They knocked at the door , and when the wife opened it and saw that it was Hansel and Grethel she said , \u201c You naughty children , why did you sleep so long in the wood ? we thought you were never coming home again ! \u201d But the father was glad , for it had gone to his heart to leave them both in the woods alone . Not very long after that there was again great scarcity in those parts , and the childr
 en heard their mother say at night in bed to their father , \u201c Everything is finished up ; we have only half a loaf , and after that the tale comes to an end . The children must be off ; we will take them farther into the wood this time , so that they shall not be able to find the way back again ; there is no other way to manage . \u201d The man felt sad at heart , and he thought , \u201c It would better to share one 's last morsel with one 's children . \u201d But the wife would listen to nothing that he said , but scolded and reproached him . He who says A must say B too , and when a man has given in once he has to do it a second time . But the children were not asleep , and had heard all the talk . When the parents had gone to sleep Hansel got up to go out and get more flint stones , as he did before , but the wife had locked the door , and Hansel could not get out ; but he comforted his little sister , and said , \u201c Do n't cry , Grethel , and go to sleep quietly , and God will help us 
 . \u201d Early the next morning the wife came and pulled the children out of bed . She gave them each a little piece of \u201c bread -less than before ; and on the way to the wood Hansel crumbled the bread in his pocket , and often stopped to throw a crumb on the ground . \u201c Hansel , what are you stopping behind and staring for ? \u201d said the father . \u201c I am looking at my little pigeon sitting on the roof , to say good-bye to me , \u201d answered Hansel . \u201c You fool , \u201d said the wife , \u201c that is no pigeon , but the morning sun shining on the chimney pots . \u201d Hansel went on as before , and strewed bread crumbs all along the road . The woman led the children far into the wood , where they had never been before in all their lives . And again there was a large fire made , and the mother said , \u201c Sit still there , you children , and when you are tired you can go to sleep ; we are going into the forest to cut wood , and in the evening , when we are ready to go home we will come and fe
 tch you . \u201d So when noon came Grethel shared her bread with Hansel , who had strewed his along the road . Then they went to sleep , and the evening passed , and no one came for the poor children . When they awoke it was dark night , and Hansel comforted his little sister , and said , \u201c Wait a little , Grethel , until the moon gets up , then we shall be able to see the way home by the crumbs of bread that I have scattered along it . \u201d So when the moon rose they got up , but they could find no crumbs of bread , for the birds of the woods and of the fields had come and picked them up . Hansel thought they might find the way all the same , but they could not . They went on all that night , and the next day from the morning until the evening , but they could not find the way out of the wood , and they were very hungry , for they had nothing to eat but the few berries they could pick up . And when they were so tired that they could no longer drag themselves along , they lay down und
 er a tree and fell asleep . It was now the third morning since they had left their father 's house . They were always trying to get back to it , but instead of that they only found themselves farther in the wood , and if help had not soon come they would have been starved . About noon they saw a pretty snow-white bird sitting on a bough , and singing so sweetly that they stopped to listen . And when he had finished the bird spread his wings and flew before them , and they followed after him until they came to a little house , and the bird perched on the roof , and when they came nearer they saw that the house was built of bread , and roofed with cakes ; and the window was of transparent sugar . \u201c We will have some of this , \u201d said Hansel , \u201c and make a fine meal . I will eat a piece of the roof , Grethel , and you can have some of the window-that will taste sweet . \u201d So Hansel reached up and broke off a bit of the roof , just to see how it tasted , and Grethel stood by the w
 indow and gnawed at it . Then they heard a thin voice call out from inside , \u201c Nibble , nibble , like a mouse , Who is nibbling at my house ? \u201d And the children answered , \u201c Never mind , It is the wind . \u201d And they went on eating , never disturbing themselves . Hansel , who found that the roof tasted very nice , took down a great piece of it , and Grethel pulled out a large round window-pane , and sat her down and began upon it . Then the door opened , and an aged woman came out , leaning upon a crutch . Hansel and Grethel felt very frightened , and let fall what they had in their hands . The old woman , however , nodded her head , and said , \u201c Ah , my dear children , how come you here ? you must come indoors and stay with me , you will be no trouble . \u201d So she took them each by the hand , and led them into her little house . And there they found a good meal laid out , of milk and pancakes , with sugar , apples , and nuts . After that she showed them two little white bed
 s , and Hansel and Grethel laid themselves down on them , and thought they were in heaven . The old woman , although her behaviour was so kind , was a wicked witch , who lay in wait for children , and had built the little house on purpose to entice them . When they were once inside she used to kill them , cook them , and eat them , and then it was a feast day with her . The witch 's eyes were red , and she could not see very far , but she had a keen scent , like the beasts , and knew very well when human creatures were near . When she knew that Hansel and Grethel were coming , she gave a spiteful laugh , and said triumphantly , \u201c I have them , and they shall not escape me ! \u201d Early in the morning , before the children were awake , she got up to look at them , and as they lay sleeping so peacefully with round rosy cheeks , she said to herself , \u201c What a fine feast I shall have ! \u201d Then she grasped Hansel with her withered hand , and led him into a little stable , and shut him
  up behind a grating ; and call and scream as he might , it was no good . Then she went back to Grethel and shook her , crying , \u201c Get up , lazy bones ; fetch water , and cook something nice for your brother ; he is outside in the stable , and must be fattened up . And when he is fat enough I will eat him . \u201d Grethel began to weep bitterly , but it was of no use , she had to do what the wicked witch bade her . And so the best kind of victuals was cooked for poor Hansel , while Grethel got nothing but crab-shells . Each morning the old woman visited the little stable , and cried , \u201c Hansel , stretch out your finger , that I may tell if you will soon be fat enough . \u201d Hansel , however , used to hold out a little bone , and the old woman , who had weak eyes , could not see what it was , and supposing it to be Hansel 's finger , wondered very much that it was not getting fatter . When four weeks had passed and Hansel seemed to remain so thin , she lost patience and could wait no
  longer . \u201c Now then , Grethel , \u201d cried she to the little girl ; \u201c be quick and draw water ; be Hansel fat or be he lean , tomorrow I must kill and cook him . \u201d Oh what a grief for the poor little sister to have to fetch water , and how the tears flowed down over her cheeks ! \u201c Dear God , pray help us ! \u201d cried she ; \u201c if we had been devoured by wild beasts in the wood at least we should have died together . \u201d \u201c Spare me your lamentations , \u201d said the old woman ; \u201c they are of no avail . \u201d Early next morning Grethel had to get up , make the fire , and fill the kettle . \u201c First we will do the baking , \u201d said the old woman ; \u201c I nave heated the oven already , and kneaded the dough . \u201d She pushed poor Grethel towards the oven , out of which the flames were already shining . \u201c Creep in , \u201d said the witch , \u201c and see if it is properly hot , so that the bread may be baked . \u201d And Grethel once in , she meant to shut the door upon her and let her be 
 baked , and then she would have eaten her . But Grethel perceived her intention , and said , \u201c I do n't know how to do it : how shall I get in ? \u201d \u201c Stupid goose , \u201d said the old woman , \u201c the opening is big enough , do you see ? I could get in myself ! \u201d and she stooped down and put her head in the oven 's mouth . Then Grethel gave her a push , so that she went in farther , and she shut the iron door upon her , and put up the bar . Oh how frightfully she howled ! but Grethel ran away , and left the wicked witch to burn miserably . Grethel went straight to Hansel , opened the stable-door , and cried , \u201c Hansel , we are free ! the old witch is dead ! \u201d Then out flew Hansel like a bird from its cage as soon as the door is opened . How rejoiced they both were ! how they fell each on the other 's neck ! and danced about , and kissed each other ! And as they had nothing more to fear they went over all the old witch 's house , and in every corner there stood chests of pear
 ls and precious stones . \u201c This is something better than flint stones , \u201d said Hansel , as he filled his pockets , and Grethel , thinking she also would like to carry something home with her , filled her apron full . i ! Now , away we go , \u201d said Hansel , \u201d if we only can get out of the witch 's wood . \u201d When they had journeyed a few hours they came to a great piece of water . \u201c We can never get across this , \u201d said Hansel , \u201c I see no stepping-stones and no bridge . \u201d \u201c And there is no boat either , \u201d said Grethel ; \u201c but here comes a white duck ; if I ask her she will help us over . \u201d So she cried , \u201c Duck , duck , here we stand , Hansel and Grethel , on the land , Stepping-stones and bridge we lack , Carry us over on your nice white back . \u201d And the duck came accordingly , and Hansel got upon her and told his sister to come too . \u201c No , \u201d answered Grethel , \u201c that would be too hard upon the duck ; we can go separately , one after the other . 
  And that was how it was managed , and after that they went on happily , until they came to the wood , and the way grew more and more familiar , till at last they saw in the distance their father 's house . Then they ran till they came up to it , rushed in at the door , and fell on their father 's neck . The man had not had a quiet hour since he left his children in the wood ; but the wife was dead . And when Grethel opened her apron the pearls and precious stones were scattered all over the room , and Hansel took one handful after another out of his pocket . Then was all care at an end , and they lived in great joy together . My tale is done , there runs a mouse , whosoever catches it , may make himself a big fur cap out of it . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/62FictHugoV_Hunchback_II5_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/62FictHugoV_Hunchback_II5_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/62FictHugoV_Hunchback_II5_EN.txt.txt
new file mode 100644
index 0000000..b4f6eb7
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/62FictHugoV_Hunchback_II5_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff CHAPTER V. RESULT OF THE DANGERS . Gringoire , thoroughly stunned by his fall , remained on the pavement in front of the Holy Virgin at the street corner . Little by little , he regained his senses ; at first , for several minutes , he was floating in a sort of half-somnolent revery , which was not without its charm , in which aeriel figures of the gypsy and her goat were coupled with Quasimodo 's heavy fist . This state lasted but a short time . A decidedly vivid sensation of cold in the part of his body which was in contact with the pavement , suddenly aroused him and caused his spirit to return to the surface . " Whence comes this chill ? " he said abruptly , to himself . He then perceived that he was lying half in the middle of the gutter . " That devil of a hunchbacked cyclops ! " he muttered between his teeth ; and he tried to rise . But he was too much dazed and bruised ; he was forced to remain where he was . Moreover , his hand was tolerably free ; he stopped up his nos
 e and resigned himself . " The mud of Paris , " he said to himself--for decidedly he thought that he was sure that the gutter would prove his refuge for the night ; and what can one do in a refuge , except dream ? --"the mud of Paris is particularly stinking ; it must contain a great deal of volatile and nitric salts . That , moreover , is the opinion of Master Nicholas Flamel , and of the alchemists-- " The word " alchemists " suddenly suggested to his mind the idea of Archdeacon Claude Frollo . He recalled the violent scene which he had just witnessed in part ; that the gypsy was struggling with two men , that Quasimodo had a companion ; and the morose and haughty face of the archdeacon passed confusedly through his memory . " That would be strange ! " he said to himself . And on that fact and that basis he began to construct a fantastic edifice of hypothesis , that card-castle of philosophers ; then , suddenly returning once more to reality , " Come ! I 'm freezing ! " he ejacula
 ted . The place was , in fact , becoming less and less tenable . Each molecule of the gutter bore away a molecule of heat radiating from Gringoire 's loins , and the equilibrium between the temperature of his body and the temperature of the brook , began to be established in rough fashion . Quite a different annoyance suddenly assailed him . A group of children , those little bare-footed savages who have always roamed the pavements of Paris under the eternal name of _gamins_ , and who , when we were also children ourselves , threw stones at all of us in the afternoon , when we came out of school , because our trousers were not torn--a swarm of these young scamps rushed towards the square where Gringoire lay , with shouts and laughter which seemed to pay but little heed to the sleep of the neighbors . They were dragging after them some sort of hideous sack ; and the noise of their wooden shoes alone would have roused the dead . Gringoire who was not quite dead yet , half raised himse
 lf . " Oh� , Hennequin Dand�che ! Oh� , Jehan Pincebourde ! " they shouted in deafening tones , " old Eustache Moubon , the merchant at the corner , has just died . We 've got his straw pallet , we 're going to have a bonfire out of it . It 's the turn of the Flemish to-day ! " And behold , they flung the pallet directly upon Gringoire , beside whom they had arrived , without espying him . At the same time , one of them took a handful of straw and set off to light it at the wick of the good Virgin . " S'death ! " growled Gringoire , " am I going to be too warm now ? " It was a critical moment . He was caught between fire and water ; he made a superhuman effort , the effort of a counterfeiter of money who is on the point of being boiled , and who seeks to escape . He rose to his feet , flung aside the straw pallet upon the street urchins , and fled . " Holy Virgin ! " shrieked the children ; " 'tis the merchant 's ghost ! " And they fled in their turn . The straw mattress remained
  master of the field . Belleforet , Father Le Juge , and Corrozet affirm that it was picked up on the morrow , with great pomp , by the clergy of the quarter , and borne to the treasury of the church of Saint Opportune , where the sacristan , even as late as 1789 , earned a tolerably handsome revenue out of the great miracle of the Statue of the Virgin at the corner of the Rue Mauconseil , which had , by its mere presence , on the memorable night between the sixth and seventh of January , 1482 , exorcised the defunct Eustache Moubon , who , in order to play a trick on the devil , had at his death maliciously concealed his soul in his straw pallet . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/63FictHugoV_Hunchback_IV3_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/63FictHugoV_Hunchback_IV3_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/63FictHugoV_Hunchback_IV3_EN.txt.txt
new file mode 100644
index 0000000..3b322ee
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/63FictHugoV_Hunchback_IV3_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff CHAPTER III . _IMMANIS PECORIS CUSTOS , IMMANIOR IPSE_ . Now , in 1482 , Quasimodo had grown up . He had become a few years previously the bellringer of Notre-Dame , thanks to his father by adoption , Claude Frollo , --who had become archdeacon of Josas , thanks to his suzerain , Messire Louis de Beaumont , --who had become Bishop of Paris , at the death of Guillaume Chartier in 1472 , thanks to his patron , Olivier Le Daim , barber to Louis XI . , king by the grace of God . So Quasimodo was the ringer of the chimes of Notre-Dame . In the course of time there had been formed a certain peculiarly intimate bond which united the ringer to the church . Separated forever from the world , by the double fatality of his unknown birth and his natural deformity , imprisoned from his infancy in that impassable double circle , the poor wretch had grown used to seeing nothing in this world beyond the religious walls which had received him under their shadow . Notre-Dame had been to him succe
 ssively , as he grew up and developed , the egg , the nest , the house , the country , the universe . There was certainly a sort of mysterious and pre-existing harmony between this creature and this church . When , still a little fellow , he had dragged himself tortuously and by jerks beneath the shadows of its vaults , he seemed , with his human face and his bestial limbs , the natural reptile of that humid and sombre pavement , upon which the shadow of the Romanesque capitals cast so many strange forms . Later on , the first time that he caught hold , mechanically , of the ropes to the towers , and hung suspended from them , and set the bell to clanging , it produced upon his adopted father , Claude , the effect of a child whose tongue is unloosed and who begins to speak . It is thus that , little by little , developing always in sympathy with the cathedral , living there , sleeping there , hardly ever leaving it , subject every hour to the mysterious impress , he came to resemble
  it , he incrusted himself in it , so to speak , and became an integral part of it . His salient angles fitted into the retreating angles of the cathedral ( if we may be allowed this figure of speech ) , and he seemed not only its inhabitant but more than that , its natural tenant . One might almost say that he had assumed its form , as the snail takes on the form of its shell . It was his dwelling , his hole , his envelope . There existed between him and the old church so profound an instinctive sympathy , so many magnetic affinities , so many material affinities , that he adhered to it somewhat as a tortoise adheres to its shell . The rough and wrinkled cathedral was his shell . It is useless to warn the reader not to take literally all the similes which we are obliged to employ here to express the singular , symmetrical , direct , almost consubstantial union of a man and an edifice . It is equally unnecessary to state to what a degree that whole cathedral was familiar to him , af
 ter so long and so intimate a cohabitation . That dwelling was peculiar to him . It had no depths to which Quasimodo had not penetrated , no height which he had not scaled . He often climbed many stones up the front , aided solely by the uneven points of the carving . The towers , on whose exterior surface he was frequently seen clambering , like a lizard gliding along a perpendicular wall , those two gigantic twins , so lofty , so menacing , so formidable , possessed for him neither vertigo , nor terror , nor shocks of amazement . To see them so gentle under his hand , so easy to scale , one would have said that he had tamed them . By dint of leaping , climbing , gambolling amid the abysses of the gigantic cathedral he had become , in some sort , a monkey and a goat , like the Calabrian child who swims before he walks , and plays with the sea while still a babe . Moreover , it was not his body alone which seemed fashioned after the Cathedral , but his mind also . In what condition 
 was that mind ? What bent had it contracted , what form had it assumed beneath that knotted envelope , in that savage life ? This it would be hard to determine . Quasimodo had been born one-eyed , hunchbacked , lame . It was with great difficulty , and by dint of great patience that Claude Frollo had succeeded in teaching him to talk . But a fatality was attached to the poor foundling . Bellringer of Notre-Dame at the age of fourteen , a new infirmity had come to complete his misfortunes : the bells had broken the drums of his ears ; he had become deaf . The only gate which nature had left wide open for him had been abruptly closed , and forever . In closing , it had cut off the only ray of joy and of light which still made its way into the soul of Quasimodo . His soul fell into profound night . The wretched being 's misery became as incurable and as complete as his deformity . Let us add that his deafness rendered him to some extent dumb . For , in order not to make others laugh , 
 the very moment that he found himself to be deaf , he resolved upon a silence which he only broke when he was alone . He voluntarily tied that tongue which Claude Frollo had taken so much pains to unloose . Hence , it came about , that when necessity constrained him to speak , his tongue was torpid , awkward , and like a door whose hinges have grown rusty . If now we were to try to penetrate to the soul of Quasimodo through that thick , hard rind ; if we could sound the depths of that badly constructed organism ; if it were granted to us to look with a torch behind those non-transparent organs to explore the shadowy interior of that opaque creature , to elucidate his obscure corners , his absurd no-thoroughfares , and suddenly to cast a vivid light upon the soul enchained at the extremity of that cave , we should , no doubt , find the unhappy Psyche in some poor , cramped , and ricketty attitude , like those prisoners beneath the Leads of Venice , who grew old bent double in a stone
  box which was both too low and too short for them . It is certain that the mind becomes atrophied in a defective body . Quasimodo was barely conscious of a soul cast in his own image , moving blindly within him . The impressions of objects underwent a considerable refraction before reaching his mind . His brain was a peculiar medium ; the ideas which passed through it issued forth completely distorted . The reflection which resulted from this refraction was , necessarily , divergent and perverted . Hence a thousand optical illusions , a thousand aberrations of judgment , a thousand deviations , in which his thought strayed , now mad , now idiotic . The first effect of this fatal organization was to trouble the glance which he cast upon things . He received hardly any immediate perception of them . The external world seemed much farther away to him than it does to us . The second effect of his misfortune was to render him malicious . He was malicious , in fact , because he was savag
 e ; he was savage because he was ugly . There was logic in his nature , as there is in ours . His strength , so extraordinarily developed , was a cause of still greater malevolence : " _Malus puer robustus_ , " says Hobbes . This justice must , however be rendered to him . Malevolence was not , perhaps , innate in him . From his very first steps among men , he had felt himself , later on he had seen himself , spewed out , blasted , rejected . Human words were , for him , always a raillery or a malediction . As he grew up , he had found nothing but hatred around him . He had caught the general malevolence . He had picked up the weapon with which he had been wounded . After all , he turned his face towards men only with reluctance ; his cathedral was sufficient for him . It was peopled with marble figures , --kings , saints , bishops , --who at least did not burst out laughing in his face , and who gazed upon him only with tranquillity and kindliness . The other statues , those of the
  monsters and demons , cherished no hatred for him , Quasimodo . He resembled them too much for that . They seemed rather , to be scoffing at other men . The saints were his friends , and blessed him ; the monsters were his friends and guarded him . So he held long communion with them . He sometimes passed whole hours crouching before one of these statues , in solitary conversation with it . If any one came , he fled like a lover surprised in his serenade . And the cathedral was not only society for him , but the universe , and all nature beside . He dreamed of no other hedgerows than the painted windows , always in flower ; no other shade than that of the foliage of stone which spread out , loaded with birds , in the tufts of the Saxon capitals ; of no other mountains than the colossal towers of the church ; of no other ocean than Paris , roaring at their bases . What he loved above all else in the maternal edifice , that which aroused his soul , and made it open its poor wings , w
 hich it kept so miserably folded in its cavern , that which sometimes rendered him even happy , was the bells . He loved them , fondled them , talked to them , understood them . From the chime in the spire , over the intersection of the aisles and nave , to the great bell of the front , he cherished a tenderness for them all . The central spire and the two towers were to him as three great cages , whose birds , reared by himself , sang for him alone . Yet it was these very bells which had made him deaf ; but mothers often love best that child which has caused them the most suffering . It is true that their voice was the only one which he could still hear . On this score , the big bell was his beloved . It was she whom he preferred out of all that family of noisy girls which bustled above him , on festival days. This bell was named Marie . She was alone in the southern tower , with her sister Jacqueline , a bell of lesser size , shut up in a smaller cage beside hers . This Jacqueline
  was so called from the name of the wife of Jean Montagu , who had given it to the church , which had not prevented his going and figuring without his head at Montfau�on . In the second tower there were six other bells , and , finally , six smaller ones inhabited the belfry over the crossing , with the wooden bell , which rang only between after dinner on Good Friday and the morning of the day before Easter . So Quasimodo had fifteen bells in his seraglio ; but big Marie was his favorite . No idea can be formed of his delight on days when the grand peal was sounded . At the moment when the archdeacon dismissed him , and said , " Go ! " he mounted the spiral staircase of the clock tower faster than any one else could have descended it . He entered perfectly breathless into the aerial chamber of the great bell ; he gazed at her a moment , devoutly and lovingly ; then he gently addressed her and patted her with his hand , like a good horse , which is about to set out on a long journey
  . He pitied her for the trouble that she was about to suffer . After these first caresses , he shouted to his assistants , placed in the lower story of the tower , to begin . They grasped the ropes , the wheel creaked , the enormous capsule of metal started slowly into motion . Quasimodo followed it with his glance and trembled . The first shock of the clapper and the brazen wall made the framework upon which it was mounted quiver . Quasimodo vibrated with the bell . " Vah ! " he cried , with a senseless burst of laughter . However , the movement of the bass was accelerated , and , in proportion as it described a wider angle , Quasimodo 's eye opened also more and more widely , phosphoric and flaming . At length the grand peal began ; the whole tower trembled ; woodwork , leads , cut stones , all groaned at once , from the piles of the foundation to the trefoils of its summit . Then Quasimodo boiled and frothed ; he went and came ; he trembled from head to foot with the tower . The
  bell , furious , running riot , presented to the two walls of the tower alternately its brazen throat , whence escaped that tempestuous breath , which is audible leagues away . Quasimodo stationed himself in front of this open throat ; he crouched and rose with the oscillations of the bell , breathed in this overwhelming breath , gazed by turns at the deep place , which swarmed with people , two hundred feet below him , and at that enormous , brazen tongue which came , second after second , to howl in his ear . It was the only speech which he understood , the only sound which broke for him the universal silence . He swelled out in it as a bird does in the sun . All of a sudden , the frenzy of the bell seized upon him ; his look became extraordinary ; he lay in wait for the great bell as it passed , as a spider lies in wait for a fly , and flung himself abruptly upon it , with might and main . Then , suspended above the abyss , borne to and fro by the formidable swinging of the bell
  , he seized the brazen monster by the ear-laps , pressed it between both knees , spurred it on with his heels , and redoubled the fury of the peal with the whole shock and weight of his body . Meanwhile , the tower trembled ; he shrieked and gnashed his teeth , his red hair rose erect , his breast heaving like a bellows , his eye flashed flames , the monstrous bell neighed , panting , beneath him ; and then it was no longer the great bell of Notre-Dame nor Quasimodo : it was a dream , a whirlwind , a tempest , dizziness mounted astride of noise ; a spirit clinging to a flying crupper , a strange centaur , half man , half bell ; a sort of horrible Astolphus , borne away upon a prodigious hippogriff of living bronze . The presence of this extraordinary being caused , as it were , a breath of life to circulate throughout the entire cathedral . It seemed as though there escaped from him , at least according to the growing superstitions of the crowd , a mysterious emanation which animat
 ed all the stones of Notre-Dame , and made the deep bowels of the ancient church to palpitate . It sufficed for people to know that he was there , to make them believe that they beheld the thousand statues of the galleries and the fronts in motion . And the cathedral did indeed seem a docile and obedient creature beneath his hand ; it waited on his will to raise its great voice ; it was possessed and filled with Quasimodo , as with a familiar spirit . One would have said that he made the immense edifice breathe . He was everywhere about it ; in fact , he multiplied himself on all points of the structure . Now one perceived with affright at the very top of one of the towers , a fantastic dwarf climbing , writhing , crawling on all fours , descending outside above the abyss , leaping from projection to projection , and going to ransack the belly of some sculptured gorgon ; it was Quasimodo dislodging the crows . Again , in some obscure corner of the church one came in contact with a s
 ort of living chimera , crouching and scowling ; it was Quasimodo engaged in thought . Sometimes one caught sight , upon a bell tower , of an enormous head and a bundle of disordered limbs swinging furiously at the end of a rope ; it was Quasimodo ringing vespers or the Angelus . Often at night a hideous form was seen wandering along the frail balustrade of carved lacework , which crowns the towers and borders the circumference of the apse ; again it was the hunchback of Notre-Dame . Then , said the women of the neighborhood , the whole church took on something fantastic , supernatural , horrible ; eyes and mouths were opened , here and there ; one heard the dogs , the monsters , and the gargoyles of stone , which keep watch night and day , with outstretched neck and open jaws , around the monstrous cathedral , barking . And , if it was a Christmas Eve , while the great bell , which seemed to emit the death rattle , summoned the faithful to the midnight mass , such an air was spread
  over the sombre fa�ade that one would have declared that the grand portal was devouring the throng , and that the rose window was watching it . And all this came from Quasimodo . Egypt would have taken him for the god of this temple ; the Middle Ages believed him to be its demon : he was in fact its soul . To such an extent was this disease that for those who know that Quasimodo has existed , Notre-Dame is to-day deserted , inanimate , dead . One feels that something has disappeared from it . That immense body is empty ; it is a skeleton ; the spirit has quitted it , one sees its place and that is all . It is like a skull which still has holes for the eyes , but no longer sight . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/64FictPoeE_Purloined_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/64FictPoeE_Purloined_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/64FictPoeE_Purloined_EN.txt.txt
new file mode 100644
index 0000000..0fe8417
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/64FictPoeE_Purloined_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff THE PURLOINED LETTER . -------------------------------------------------------------------------------- Nil sapientiae odiosius acumine nimio . Seneca . At Paris , just after dark one gusty evening in the autumn of 18 \u2014 , I was enjoying the twofold luxury of meditation and a meerschaum , in company with my friend C. Auguste Dupin , in his little back library , or book-closet , au troisi�me , No. 33 , Rue Dun�t , Faubourg St. Germain . For one hour at least we had maintained a profound silence ; while each , to any casual observer , might have seemed intently and exclusively occupied with the curling eddies of smoke that oppressed the atmosphere of the chamber . For myself , however , I was mentally discussing certain topics which had formed matter for conversation between us at an earlier period of the evening ; I mean the affair of the Rue Morgue , and the mystery attending the murder of Marie Rog�t . I looked upon it , therefore , as something of a coincidence , when the 
 door of our apartment was thrown open and admitted our old acquaintance , Monsieur G \u2014 \u2014 , the Prefect of the Parisian police . We gave him a hearty welcome ; for there was nearly half as much of the entertaining as of the contemptible about the man , and we had not seen him for several years . We had been sitting in the dark , and Dupin now arose for the purpose of lighting a lamp , but sat down again , without doing so , upon G. 's saying that he had called to consult us , or rather to ask the opinion of my friend , about some official business which had occasioned a great deal of trouble . " If it is any point requiring reflection , " observed Dupin , as he forebore to enkindle the wick , " we shall examine it to better purpose in the dark . " " That is another of your odd notions , " said the Prefect , who had a fashion of calling every thing " odd " that was beyond his comprehension , and thus lived amid an absolute legion of " oddities . " " Very true , " said Dupin , as h
 e supplied his visiter with a pipe , and rolled towards him a comfortable chair . " And what is the difficulty now ? " I asked . " Nothing more in the assassination way , I hope ? " " Oh no ; nothing of that nature . The fact is , the business is very simple indeed , and I make no doubt that we can manage it sufficiently well ourselves ; but then I thought Dupin would like to hear the details of it , because it is so excessively odd . " " Simple and odd , " said Dupin . " Why , yes ; and not exactly that , either . The fact is , we have all been a good deal puzzled because the affair is so simple , and yet baffles us altogether . " " Perhaps it is the very simplicity of the thing which puts you at fault , " said my friend . " What nonsense you do talk ! " replied the Prefect , laughing heartily . " Perhaps the mystery is a little too plain , " said Dupin . " Oh , good heavens ! who ever heard of such an idea ? " " A little too self evident . " " Ha ! ha ! ha ! \u2014 ha ! ha ! ha ! \u2014
  ho ! ho ! ho ! " roared our visiter , profoundly amused , " oh , Dupin , you will be the death of me yet ! " " And what , after all , is the matter on hand ? " I asked . " Why , I will tell you , " replied the Prefect , as he gave a long , steady and contemplative puff , and settled himself in his chair . " I will tell you in a few words ; but , before I begin , let me caution you that this is an affair demanding the greatest secrecy , and that I should most probably lose the position I now hold , were it known that I confided it to any one . " " Proceed , " said I. " Or not , " said Dupin . " Well , then ; I have received personal information , from a very high quarter , that a certain document of the last importance , has been purloined from the royal apartments . The individual who purloined it is known ; this beyond a doubt ; he was seen to take it . It is known , also , that it still remains in his possession . " " How is this known ? " asked Dupin . " It is clearly inferred ,
  " replied the Prefect , " from the nature of the document , and from the non-appearance of certain results which would at once arise from its passing out of the robber 's possession ; \u2014 that is to say , from his employing it as he must design in the end to employ it . " " Be a little more explicit , " I said . " Well , I may venture so far as to say that the paper gives its holder a certain power in a certain quarter where such power is immensely valuable . " The Prefect was fond of the cant of diplomacy . " Still I do not quite understand , " said Dupin . " No ? Well ; the disclosure of the document to a third person , who shall be nameless , would bring in question the honor of a personage of most exalted station ; and this fact gives the holder of the document an ascendancy over the illustrious personage whose honor and peace are so jeopardized . " " But this ascendancy , " I interposed , " would depend upon the robber 's knowledge of the loser 's knowledge of the robber . Who
  would dare \u2014 " " The thief , " said G. , " is the Minister D \u2014 \u2014 , who dares all things , those unbecoming as well as those becoming a man . The method of the theft was not less ingenious than bold . The document in question\u2014a letter , to be frank\u2014had been received by the personage robbed while alone in the royal boudoir . During its perusal she was suddenly interrupted by the entrance of the other exalted personage from whom especially it was her wish to conceal it . After a hurried and vain endeavor to thrust it in a drawer , she was forced to place it , open as it was , upon a table . The address , however , was uppermost , and , the contents thus unexposed , the letter escaped notice . At this juncture enters the Minister D \u2014 \u2014 . His lynx eye immediately perceives the paper , recognises the handwriting of the address , observes the confusion of the personage addressed , and fathoms her secret . After some business transactions , hurried through in his ordinary man
 ner , he produces a letter somewhat similar to the one in question , opens it , pretends to read it , and then places it in close juxtaposition to the other . Again he converses , for some fifteen minutes , upon the public affairs . At length , in taking leave , he takes also from the table the letter to which he had no claim . Its rightful owner saw , but , of course , dared not call attention to the act , in the presence of the third personage who stood at her elbow . The minister decamped ; leaving his own letter\u2014one of no importance\u2014upon the table . " " Here , then , " said Dupin to me , " you have precisely what you demand to make the ascendancy complete\u2014the robber 's knowledge of the loser 's knowledge of the robber . " " Yes , " replied the Prefect ; " and the power thus attained has , for some months past , been wielded , for political purposes , to a very dangerous extent . The personage robbed is more thoroughly convinced , every day , of the necessity of reclaiming 
 her letter . But this , of course , cannot be done openly . In fine , driven to despair , she has committed the matter to me . " " Than whom , " said Dupin , amid a perfect whirlwind of smoke , " no more sagacious agent could , I suppose , be desired , or even imagined . " " You flatter me , " replied the Prefect ; " but it is possible that some such opinion may have been entertained . " " It is clear , " said I , " as you observe , that the letter is still in possession of the minister ; since it is this possession , and not any employment of the letter , which bestows the power . With the employment the power departs . " " True , " said G. ; " and upon this conviction I proceeded . My first care was to make thorough search of the minister 's hotel ; and here my chief embarrassment lay in the necessity of searching without his knowledge . Beyond all things , I have been warned of the danger which would result from giving him reason to suspect our design . " " But , " said I , " you
  are quite au fait in these investigations . The Parisian police have done this thing often before . " " O yes ; and for this reason I did not despair . The habits of the minister gave me , too , a great advantage . He is frequently absent from home all night . His servants are by no means numerous . They sleep at a distance from their master 's apartment , and , being chiefly Neapolitans , are readily made drunk . I have keys , as you know , with which I can open any chamber or cabinet in Paris . For three months a night has not passed , during the greater part of which I have not been engaged , personally , in ransacking the D \u2014 \u2014 Hotel . My honor is interested , and , to mention a great secret , the reward is enormous . So I did not abandon the search until I had become fully satisfied that the thief is a more astute man than myself . I fancy that I have investigated every nook and corner of the premises in which it is possible that the paper can be concealed . " " But is it 
 not possible , " I suggested , " that although the letter may be in possession of the minister , as it unquestionably is , he may have concealed it elsewhere than upon his own premises ? " " This is barely possible , " said Dupin . " The present peculiar condition of affairs at court , and especially of those intrigues in which D \u2014 \u2014 is known to be involved , would render the instant availability of the document\u2014its susceptibility of being produced at a moment 's notice\u2014a point of nearly equal importance with its possession . " " Its susceptibility of being produced ? " said I. " That is to say , of being destroyed , " said Dupin . " True , " I observed ; " the paper is clearly then upon the premises . As for its being upon the person of the minister , we may consider that as out of the question . " " Entirely , " said the Prefect . " He has been twice waylaid , as if by footpads , and his person rigorously searched under my own inspection . " " You might have spared yoursel
 f this trouble , " said Dupin . " D \u2014 \u2014 , I presume , is not altogether a fool , and , if not , must have anticipated these waylayings , as a matter of course . " " Not altogether a fool , " said G. , " but then he 's a poet , which I take to be only one remove from a fool . " " True , " said Dupin , after a long and thoughtful whiff from his meerschaum , " although I have been guilty of certain doggrel myself . " " Suppose you detail , " said I , " the particulars of your search . " " Why the fact is , we took our time , and we searched every where . I have had long experience in these affairs . I took the entire building , room by room ; devoting the nights of a whole week to each . We examined , first , the furniture of each apartment . We opened every possible drawer ; and I presume you know that , to a properly trained police agent , such a thing as a secret drawer is impossible . Any man is a dolt who permits a 'secret ' drawer to escape him in a search of this kind . The 
 thing is so plain . There is a certain amount of bulk\u2014of space\u2014to be accounted for in every cabinet . Then we have accurate rules . The fiftieth part of a line could not escape us . After the cabinets we took the chairs . The cushions we probed with the fine long needles you have seen me employ . From the tables we removed the tops . " " Why so ? " " Sometimes the top of a table , or other similarly arranged piece of furniture , is removed by the person wishing to conceal an article ; then the leg is excavated , the article deposited within the cavity , and the top replaced . The bottoms and tops of bedposts are employed in the same way . " " But could not the cavity be detected by sounding ? " I asked . " By no means , if , when the article is deposited , a sufficient wadding of cotton be placed around it . Besides , in our case , we were obliged to proceed without noise . " " But you could not have removed\u2014you could not have taken to pieces all articles of furniture in which
  it would have been possible to make a deposit in the manner you mention . A letter may be compressed into a thin spiral roll , not differing much in shape or bulk from a large knitting needle , and in this form it might be inserted into the rung of a chair , for example . You did not take to pieces all the chairs ? " " Certainly not ; but we did better\u2014we examined the rungs of every chair in the hotel , and , indeed the jointings of every description of furniture , by the aid of a most powerful microscope . Had there been any traces of recent disturbance we should not have failed to detect it instantly . A single grain of gimlet dust , for example , would have been as obvious as an apple . Any disorder in the glueing\u2014any unusual gaping in the joints\u2014would have sufficed to insure detection . " " I presume you looked to the mirrors , between the boards and the plates , and you probed the beds and the bed clothes , as well as the curtains and carpets . " " That of course ; and w
 hen we had absolutely completed every particle of the furniture in this way , then we examined the house itself . We divided its entire surface into compartments , which we numbered , so that none might be missed ; then we scrutinized each individual square inch throughout the premises , including the two houses immediately adjoining , with the microscope , as before . " " The two houses adjoining ! " I exclaimed ; " you must have had a great deal of trouble . " " We had ; but the reward offered is prodigious ! " " You include the grounds about the houses ? " " All the grounds are paved with brick . They gave us comparatively little trouble . We examined the moss between the bricks , and found it undisturbed . " " You looked among D\u2014\u2014 's papers , of course , and into the books of the library ? " " Certainly ; we opened every package and parcel ; we not only opened every book , but we turned over every leaf in each volume , not contenting ourselves with a mere shake , according t
 o the fashion of some of our police officers . We also measured the thickness of every book cover , with the most accurate admeasurement , and applied to each the most jealous scrutiny of the microscope . Had any of the bindings been recently meddled with , it would have been utterly impossible that the fact should have escaped observation . Some five or six volumes , just from the hands of the binder , we carefully probed , longitudinally , with the needles . " " You explored the floors beneath the carpets ? " " Beyond doubt . We removed every carpet , and examined the boards with the microscope . " " And the paper on the walls ? " " Yes . " " You looked into the cellars ? " " We did . " " Then , " I said , " you have been making a miscalculation , and the letter is not upon the premises , as you suppose . " " I fear you are right there , " said the Prefect . " And now , Dupin , what would you advise me to do ? " " To make a thorough re-search of the premises . " " That is absolute
 ly needless , " replied G \u2014 \u2014 . " I am not more sure that I breathe than I am that the letter is not at the Hotel . " " I have no better advice to give you , " said Dupin . " You have , of course , an accurate description of the letter ? " " Oh yes ! " \u2014 And here the Prefect , producing a memorandum book proceeded to read aloud a minute account of the internal , and especially of the external appearance of the missing document . Soon after finishing the perusal of this description , he took his departure , more entirely depressed in spirits than I had ever known the good gentleman before . In about a month afterwards he paid us another visit , and found us occupied very nearly as before . He took a pipe and a chair and entered into some ordinary conversation . At length I said , \u2014 " Well , but G \u2014 \u2014 , what of the purloined letter ? I presume you have at last made up your mind that there is no such thing as overreaching the Minister ? " " Confound him , say I\u2014yes ; I ma
 de the re-examination , however , as Dupin suggested-but it was all labor lost , as I knew it would be . " " How much was the reward offered , did you say ? " asked Dupin . " Why , a very great deal\u2014a very liberal reward\u2014I do n't like to say how much , precisely ; but one thing I will say , that I would n't mind giving my individual check for fifty thousand francs to any one who could obtain me that letter . The fact is , it is becoming of more and more importance every day ; and the reward has been lately doubled . If it were trebled , however , I could do no more than I have done . " " Why , yes , " said Dupin , drawlingly , between the whiffs of his meerschaum , " I really-think , G \u2014 , you have not exerted yourself\u2014to the utmost in this matter . You might\u2014do a little more , I think , eh ? " " How ? \u2014 in what way ? ' " Why\u2014puff , puff\u2014you might\u2014puff , puff\u2014employ counsel in the matter , eh ? \u2014 puff , puff , puff . Do you remember the story they tell of Abern
 ethy ? " " No ; hang Abernethy ! " " To be sure ! hang him and welcome . But , once upon a time , a certain rich miser conceived the design of spunging upon this Abernethy for a medical opinion . Getting up , for this purpose , an ordinary conversation in a private company , he insinuated his case to the physician , as that of an imaginary individual . " 'We will suppose , ' said the miser , 'that his symptoms are such and such ; now , doctor , what would you have directed him to take ? ' " 'Take ! ' said Abernethy , 'why , take advice , to be sure . ' " " But , " said the Prefect , a little discomposed , " I am perfectly willing to take advice , and to pay for it . I would really give fifty thousand francs to any one who would aid me in the matter . " " In that case , " replied Dupin , opening a drawer , and producing a check book , " you may as well fill me up a check for the amount mentioned . When you have signed it , I will hand you the letter . " I was astounded . The Prefect 
 appeared absolutely thunder stricken . For some minutes he remained speechless and motionless , looking incredulously at my friend with open mouth , and eyes that seemed starting from their sockets ; then , apparently recovering himself in some measure , he seized a pen , and after several pauses and vacant stares , finally filled up and signed a check for fifty thousand francs , and handed it across the table to Dupin . The latter examined it carefully and deposited it in his pocket book ; then , unlocking an escritoire , took thence a letter and gave it to the Prefect . This functionary grasped it in a perfect agony of joy , opened it with a trembling hand , cast a rapid glance at its contents , and then , scrambling and struggling to the door , rushed at length unceremoniously from the room and from the house , without having uttered a syllable since Dupin had requested him to fill up the check . When he had gone , my friend entered into some explanations . " The Parisian police 
 , " he said , " are exceedingly able in their way . They are persevering , ingenious , cunning , and thoroughly versed in the knowledge which their duties seem chiefly to demand . Thus , when G \u2014 \u2014 detailed to us his mode of searching the premises at the Hotel D \u2014 \u2014 , I felt entire confidence in his having made a satisfactory investigation\u2014so far as his labors extended . " " So far as his labors extended ? " said I. " Yes , " said Dupin . " The measures adopted were not only the best of their kind , but carried out to absolute perfection . Had the letter been deposited within the range of their search , these fellows would , beyond a question , have found it . " I merely laughed\u2014but he seemed quite serious in all that he said . " The measures , then , " he continued , " were good in their kind , and well executed ; their defect lay in their being inapplicable to the case , and to the man . A certain set of highly ingenious resources are , with the Prefect , a sort of Pro
 crustean bed , to which he forcibly adapts his designs . But he perpetually errs by being too deep or too shallow , for the matter in hand ; and many a schoolboy is a better reasoner than he . I knew one about eight years of age , whose success at guessing in the game of 'even and odd ' attracted universal admiration . This game is simple , and is played with marbles . One player holds in his hand a number of these toys , and demands of another whether that number is even or odd . If the guess is right , the guesser wins one ; if wrong , he loses one . The boy to whom I allude won all the marbles of the school . Of course he had some principle of guessing ; and this lay in mere observation and admeasurement of the astuteness of his opponents . For example , an arrant simpleton is his opponent , and , holding up his closed hand , asks , 'are they even or odd ? ' Our schoolboy replies , 'odd , ' and loses ; but upon the second trial he wins , for he then says to himself , 'the simplet
 on had them even upon the first trial , and his amount of cunning is just sufficient to make him have them odd upon the second ; I will therefore guess odd ; '\u2014he guesses odd , and wins . Now , with a simpleton a degree above the first , he would have reasoned thus : 'This fellow finds that in the first instance I guessed odd , and , in the second , he will propose to himself , upon the first impulse , a simple variation from even to odd , as did the first simpleton ; but then a second thought will suggest that this is too simple a variation , and finally he will decide upon putting it even as before . I will therefore guess even ; '\u2014he guesses even , and wins . Now this mode of reasoning in the schoolboy , whom his fellows termed 'lucky , '\u2014what , in its last analysis , is it ? " " It is merely , " I said , " an identification of the reasoner 's intellect with that of his opponent . " " It is , " said Dupin ; " and , upon inquiring , of the boy by what means he effected the t
 horough identification in which his success consisted , I received answer as follows : 'When I wish to find out how wise , or how stupid , or how good , or how wicked is any one , or what are his thoughts at the moment , I fashion the expression of my face , as accurately as possible , in accordance with the expression of his , and then wait to see what thoughts or sentiments arise in my mind or heart , as if to match or correspond with the expression . ' This response of the schoolboy lies at the bottom of all the spurious profundity which has been attributed to Rochefoucault , to La Bougive , to Machiavelli , and to Campanella . " " And the identification , " I said , " of the reasoner 's intellect with that of his opponent , depends , if I understand you aright , upon the accuracy with which the opponent 's intellect is admeasured . " " For its practical value it depends upon this , " replied Dupin ; " and the Prefect and his cohort fail so frequently , first , by default of this
  identification , and , secondly , by ill-admeasurement , or rather through non-admeasurement , of the intellect with which they are engaged . They consider only their own ideas of ingenuity ; and , in searching for anything hidden , advert only to the modes in which they would have hidden it . They are right in this much\u2014that their own ingenuity is a faithful representative of that of the mass ; but when the cunning of the individual felon is diverse in character from their own , the felon foils them , of course . This always happens when it is above their own , and very usually when it is below . They have no variation of principle in their investigations ; at best , when urged by some unusual emergency\u2014by some extraordinary reward\u2014they extend or exaggerate their old modes of practice , without touching their principles . What , for example , in this case of D \u2014 , has been done to vary the principle of action ? What is all this boring , and probing , and sounding , and scr
 utinizing with the microscope and dividing the surface of the building into registered square inches\u2014what is it all but an exaggeration of the application of the one principle or set of principles of search , which are based upon the one set of notions regarding human ingenuity , to which the Prefect , in the long routine of his duty , has been accustomed ? Do you not see he has taken it for granted that all men proceed to conceal a letter , \u2014 not exactly in a gimlet hole bored in a chair leg\u2014but , at least , in some out-of-the-way hole or corner suggested by the same tenor of thought which would urge a man to secrete a letter in a gimlet hole bored in a chair leg ? And do you not see also , that such recherch�s nooks for concealment are adapted only for ordinary occasions , and would be adopted only by ordinary intellects ; for , in all cases of concealment , a disposal of the article concealed\u2014a disposal of it in this recherch� manner , \u2014 is , in the very first instanc
 e , presumable and presumed ; and thus its discovery depends , not at all upon the acumen , but altogether upon the mere care , patience , and determination of the seekers ; and where the case is of importance\u2014or , what amounts to the same thing in the policial eyes , when the reward is of magnitude , \u2014 the qualities in question have never been known to fail . You will now understand what I meant in suggesting that , had the purloined letter been hidden any where within the limits of the Prefect 's examination\u2014in other words , had the principle of its concealment been comprehended within the principles of the Prefect\u2014its discovery would have been a matter altogether beyond question . This functionary , however , has been thoroughly mystified ; and the remote source of his defeat lies in the supposition that the Minister is a fool , because he has acquired renown as a poet . All fools are poets ; this the Prefect feels ; and he is merely guilty of a non distributio medii in t
 hence inferring that all poets are fools . " " But is this really the poet ? " I asked . " There are two brothers , I know ; and both have attained reputation in letters . The Minister I believe has written learnedly on the Differential Calculus . He is a mathematician , and no poet . " " You are mistaken ; I know him well ; he is both . As poet and mathematician , he would reason well ; as mere mathematician , he could not have reasoned at all , and thus would have been at the mercy of the Prefect . " " You surprise me , " I said , " by these opinions , which have been contradicted by the voice of the world . You do not mean to set at naught the well-digested idea of centuries . The mathematical reason has long been regarded as the reason par excellence . " " 'Il y a � pari�r , ' " replied Dupin , quoting from Chamfort , " 'que toute id�e publique , toute convention re�ue est une sottise , car elle a convenue au plus grand nombre . ' The mathematicians , I grant you , have done
  their best to promulgate the popular error to which you allude , and which is none the less an error for its promulgation as truth . With an art worthy a better cause , for example , they have insinuated the term 'analysis ' into application to algebra . The French are the originators of this particular deception ; but if a term is of any importance\u2014if words derive any value from applicability\u2014then 'analysis ' conveys 'algebra ' about as much as , in Latin , 'ambitus ' implies 'ambition , ' 'religio ' 'religion , ' or 'homines honesti , ' a set of honorable men . " " You have a quarrel on hand , I see , " said I , " with some of the algebraists of Paris ; but proceed . " " I dispute the availability , and thus the value , of that reason which is cultivated in any especial form other than the abstractly logical . I dispute , in particular , the reason educed by mathematical study . The mathematics are the science of form and quantity ; mathematical reasoning is merely logic appl
 ied to observation upon form and quantity . The great error lies in supposing that even the truths of what is called pure algebra , are abstract or general truths . And this error is so egregious that I am confounded at the universality with which it has been received . Mathematical axioms are not axioms of general truth . What is true of relation\u2014of form and quantity\u2014is often grossly false in regard to morals , for example . In this latter science it is very usually untrue that the aggregated parts are equal to the whole . In chemistry also the axiom fails . In the consideration of motive it fails ; for two motives , each of a given value , have not , necessarily , a value when united , equal to the sum of their values apart . There are numerous other mathematical truths which are only truths within the limits of relation . But the mathematician argues , from his finite truths , through habit , as if they were of an absolutely general applicability\u2014as the world indeed imagine
 s them to be . Bryant , in his very learned 'Mythology , ' mentions an analogous source of error , when he says that 'although the Pagan fables are not believed , yet we forget ourselves continually , and make inferences from them as existing realities . ' With the algebraists , however , who are Pagans themselves , the 'Pagan fables ' are believed , and the inferences are made , not so much through lapse of memory , as through an unaccountable addling of the brains . In short , I never yet encountered the mere mathematician who could be trusted out of equal roots , or one who did not clandestinely hold it as a point of his faith that x2+px was absolutely and unconditionally equal to q. Say to one of these gentlemen , by way of experiment , if you please , that you believe occasions may occur where x2+px is not altogether equal to q , and , having made him understand what you mean , get out of his reach as speedily as convenient , for , beyond doubt , he will endeavor to knock you d
 own . " I mean to say , " continued Dupin , while I merely laughed at his last observations , " that if the Minister had been no more than a mathematician , the Prefect would have been under no necessity of giving me this check . I know him , however , as both mathematician and poet , and my measures were adapted to his capacity , with reference to the circumstances by which he was surrounded . I knew him as a courtier , too , and as a bold intriguant . Such a man , I considered , could not fail to be aware of the ordinary policial modes of action . He could not have failed to anticipate\u2014and events have proved that he did not fail to anticipate\u2014the waylayings to which he was subjected . He must have foreseen , I reflected , the secret investigations of his premises . His frequent absences from home at night , which were hailed by the Prefect as certain aids to his success , I regarded only as ruses , to afford opportunity for thorough search to the police , and thus the sooner t
 o impress them with the conviction to which G \u2014 \u2014 , in fact , did finally arrive\u2014the conviction that the letter was not upon the premises . I felt , also , that the whole train of thought , which I was at some pains in detailing to you just now , concerning the invariable principle of policial action in searches for articles concealed\u2014I felt that this whole train of thought would necessarily pass through the mind of the Minister . It would imperatively lead him to despise all the ordinary nooks of concealment . He could not , I reflected , be so weak as not to see that the most intricate and remote recess of his hotel would be as open as his commonest closets to the eyes , to the probes , to the gimlets , and to the microscopes of the Prefect . I saw , in fine , that he would be driven , as a matter of course , to simplicity , if not deliberately induced to it as a matter of choice . You will remember , perhaps , how desperately the Prefect laughed when I suggested , upon ou
 r first interview , that it was just possible this mystery troubled him so much on account of its being so very self-evident . " " Yes , " said I , " I remember his merriment well . I really thought he would have fallen into convulsions . " " The material world , " continued Dupin , " abounds with very strict analogies to the immaterial ; and thus some color of truth has been given to the rhetorical dogma , that metaphor , or simile , may be made to strengthen an argument , as well as to embellish a description . The principle of the vis inerti� , for example , seems to be identical in physics and metaphysics . It is not more true in the former , that a large body is with more difficulty set in motion than a smaller one , and that its subsequent momentum is commensurate with this difficulty , than it is , in the latter , that intellects of the vaster capacity , while more forcible , more constant , and more eventful in their movements than those of inferior grade , are yet the less
  readily moved , and more embarrassed and full of hesitation in the first few steps of their progress . Again : have you ever noticed which of the street signs , over the shop-doors , are the most attractive of attention ? " " I have never given the matter a thought , " I said . " There is a game of puzzles , " he resumed , " which is played upon a map . One party playing requires another to find a given word\u2014the name of town , river , state or empire\u2014any word , in short , upon the motley and perplexed surface of the chart . A novice in the game generally seeks to embarrass his opponents by giving them the most minutely lettered names ; but the adept selects such words as stretch , in large characters , from one end of the chart to the other . These , like the over largely lettered signs and placards of the street , escape observation by dint of being excessively obvious ; and here the physical oversight is precisely analogous with the moral inapprehension by which the intellect
  suffers to pass unnoticed those considerations which are too obtrusively and too palpably self evident . But this is a point , it appears , somewhat above or beneath the understanding of the Prefect . He never once thought it probable , or possible , that the Minister had deposited the letter immediately beneath the nose of the whole world , by way of best preventing any portion of that world from perceiving it . " But the more I reflected upon the daring , dashing , and discriminating ingenuity of D \u2014 \u2014 ; upon the fact that the document must always have been at hand , if he intended to use it to good purpose ; and upon the decisive evidence , obtained by the Prefect , that it was not hidden within the limits of that dignitary 's ordinary search\u2014the more satisfied I became that , to conceal this letter , the Minister had resorted to the comprehensive and sagacious expedient of not attempting to conceal it at all . " Full of these ideas , I prepared myself with a pair of green
  spectacles , and called one fine morning , quite by accident , at the Ministerial hotel . I found D \u2014 \u2014 at home , yawning , lounging , and dawdling , as usual , and pretending to be in the last extremity of ennui . He is , perhaps , the most really energetic human being now alive\u2014but that is only when nobody sees him . " To be even with him , I complained of my weak eyes , and lamented the necessity of the spectacles , under cover of which I cautiously and thoroughly surveyed the whole apartment , while seemingly intent only upon the conversation of my host . " I paid especial attention to a large writing-table near which he sat , and upon which lay confusedly , some miscellaneous letters and other papers , with one or two musical instruments and a few books . Here , however , after a long and very deliberate scrutiny , I saw nothing to excite particular suspicion . " At length my eyes , in going the circuit of the room , fell upon a trumpery fillagree card-rack of pasteboard
  , that hung dangling by a dirty blue ribbon , from a little brass knob just beneath the middle of the mantel-piece . In this rack , which had three or four compartments , were five or six visiting cards and a solitary letter . This last was much soiled and crumpled . It was torn nearly in two , across the middle\u2014as if a design , in the first instance , to tear it entirely up as worthless , had been altered , or stayed , in the second . It had a large black seal , bearing the D \u2014 \u2014 cipher very conspicuously , and was addressed , in a diminutive female hand , to D \u2014 \u2014 , the minister , himself . It was thrust carelessly , and even , as it seemed , contemptuously , into one of the uppermost divisions of the rack . " No sooner had I glanced at this letter , than I concluded it to be that of which I was in search . To be sure , it was , to all appearance , radically different from the one of which the Prefect had read us so minute a description . Here the seal was large and bla
 ck , with the D \u2014 \u2014 cipher ; there it was small and red , with the ducal arms of the S \u2014 \u2014 family . Here , the address , to the Minister , diminutive and feminine ; there the superscription , to a certain royal personage , was markedly bold and decided ; the size alone formed a point of correspondence . But , then , the radicalness of these differences , which was excessive ; the dirt ; the soiled and torn condition of the paper , so inconsistent with the true methodical habits of D \u2014 \u2014 , and so suggestive of a design to delude the beholder into an idea of the worthlessness of the document ; these things , together with the hyper-obtrusive situation of this document , full in the view of every visiter , and thus exactly in accordance with the conclusions to which I had previously arrived ; these things , I say , were strongly corroborative of suspicion , in one who came with the intention to suspect . " I protracted my visit as long as possible , and , while I maintained
  a most animated discussion with the Minister upon a topic which I knew well had never failed to interest and excite him , I kept my attention really riveted upon the letter . In this examination , I committed to memory its external appearance and arrangement in the rack ; and also fell , at length , upon a discovery which set at rest whatever trivial doubt I might have entertained . In scrutinizing the edges of the paper , I observed them to be more chafed than seemed necessary . They presented the broken appearance which is manifested when a stiff paper , having been once folded and pressed with a folder , is refolded in a reversed direction , in the same creases or edges which had formed the original fold . This discovery was sufficient . It was clear to me that the letter had been turned , as a glove , inside out , re-directed , and re-sealed . I bade the Minister good morning , and took my departure at once , leaving a gold snuff box upon the table . " The next morning I called
  for the snuff-box , when we resumed , quite eagerly , the conversation of the preceding day . While thus engaged , however , a loud report , as if of a pistol , was heard immediately beneath the windows of the hotel , and was succeeded by a series of fearful screams , and the shoutings of a terrified mob . D \u2014 \u2014 rushed to a casement , threw it open , and looked out . In the meantime , I stepped to the card-rack took the letter , put it in my pocket , and replaced it by a fac-simile , ( so far as regards externals , ) which I had carefully prepared at my lodgings\u2014imitating the D \u2014 \u2014 cipher , very readily , by means of a seal formed of bread . " The disturbance in the street had been occasioned by the frantic behavior of a man with a musket . He had fired it among a crowd of women and children . It proved , however , to have been without ball , and the fellow was suffered to go his way as a lunatic or a drunkard . When he had gone , D \u2014 \u2014 came from the window , whither 
 I had followed him immediately upon securing the object in view . Soon afterwards I bade him farewell . The pretended lunatic was a man in my own pay . " " But what purpose had you , " I asked , " in replacing the letter by a fac-simile ? Would it not have been better , at the first visit , to have seized it openly , and departed ? " " D \u2014 \u2014 , " replied Dupin , " is a desperate man , and a man of nerve . His hotel , too , is not without attendants devoted to his interests . Had I made the wild attempt you suggest , I might never have left the Ministerial presence alive . The good people of Paris might have heard of me no more . But I had an object apart from these considerations . You know my political prepossessions . In this matter , I act as a partisan of the lady concerned . For eighteen months the Minister has had her in his power . She has now him in hers\u2014since , being unaware that the letter is not in his possession , he will proceed with his exactions as if it was . Th
 us will he inevitably commit himself , at once , to his political destruction . His downfall , too , will not be more precipitate than awkward . It is all very well to talk about the facilis descensus Averni ; but in all kinds of climbing , as Catalani said of singing , it is far more easy to get up than to come down . In the present instance I have no sympathy\u2014at least no pity\u2014for him who descends . He is that monstrum horrendum , an unprincipled man of genius . I confess , however , that I should like very well to know the precise character of his thoughts , when , being defied by her whom the Prefect terms 'a certain personage ' he is reduced to opening the letter which I left for him in the card-rack . " " How ? did you put any thing particular in it ? " " Why\u2014it did not seem altogether right to leave the interior blank\u2014that would have been insulting . D \u2014 \u2014 , at Vienna once , did me an evil turn , which I told him , quite good humoredly , that I should remember . So
  , as I knew he would feel some curiosity in regard to the identity of the person who had outwitted him , I thought it a pity not to give him a clue . He is well acquainted with my MS . , and I just copied into the middle of the blank sheet the words \u2014 " ' \u2014 \u2014 \u2014 \u2014 Un dessein si funeste , S'il n'est digne d'Atr�e , est digne de Thyeste . They are to be found in Cr�billon 's 'Atr�e . ' " 
\ No newline at end of file


[50/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/apps/object_dedup/SimilarityAccessorBase.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/apps/object_dedup/SimilarityAccessorBase.java b/opennlp-similarity/src/main/java/opennlp/tools/apps/object_dedup/SimilarityAccessorBase.java
new file mode 100755
index 0000000..c8156ea
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/apps/object_dedup/SimilarityAccessorBase.java
@@ -0,0 +1,739 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.apps.object_dedup;
+
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.regex.Pattern;
+
+import opennlp.tools.similarity.apps.BingQueryRunner;
+import opennlp.tools.similarity.apps.HitBase;
+import opennlp.tools.similarity.apps.utils.LevensteinDistanceFinder;
+import opennlp.tools.similarity.apps.utils.StringDistanceMeasurer;
+import opennlp.tools.similarity.apps.utils.Utils;
+import opennlp.tools.textsimilarity.TextProcessor;
+
+import org.apache.commons.lang.StringUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/* This is a template class for deduplicator */
+
+public class SimilarityAccessorBase
+{
+	private static final Logger LOG = LoggerFactory.getLogger(SimilarityAccessorBase.class);
+
+	public static final int MAX_EV_TO_RECOMM = 6;
+
+	private List<String> namesBothSides;
+
+	protected static final String[] englishPrepositions = new String[] { "a", "aboard", "about", "above", "absent",
+		"across", "after", "against", "along", "alongside", "among", "around", "as", "at", "before", "behind", "below",
+		"beneath", "between", "beyond", "but", "by", "despite", "down", "during", "except", "excluding", "failing",
+		"following", "for", "from", "in", "including", "inside", "into", "like", "near", "next", "of", "off", "on",
+		"onto", "only", "opposite", "out", "outside", "over", "pace", "past", "per", "since", "than", "through", "and",
+		"thru", "till", "to", "toward", "under", "up", "upon", "versus", "with", "within", "you", "must", "know",
+		"when" };
+
+	protected List<String> commonWordsInEventTitles = Arrays.asList(new String[] { "community", "party", "film",
+		"music", "exhibition", "kareoke", "guitar", "quartet", "reggae", "r&b", "band", "dj ", "piano", "pray",
+		"worship", "god", "training", "class", "development", "training", "class", "course", "our", "comedy", ",fun",
+		"musical", "group", "alliance", "session", "feeding", "introduction", "school", "conversation", "learning",
+		"nursery", "unity", "trivia", "chat", "conference", "tuition", "technology", "teen", "communication",
+		"reception", "management", "beginner", "beginning", "collabora", "reuninon", "political", "course", "age",
+		"ages", "through", "grade", "networking", "workshop", "demonstration", "tuning", "program", "summit",
+		"convention", "day", "night", "one", "two", "outfest", "three", "online", "writing", "seminar", "coach",
+		",expo", "advanced", "beginner", "intermediate", "earn", "free", "ii", "iii", "skills", "skill", "artist",
+		"summer", "winter", "autumn", "spring", "camp", "vacation", "miscrosoft", "kid", "child", "kids", "children",
+		"every", "everyone", "dancer", "dancers", "senior", "seniors", "basic", "elementary", "outfest", "2008",
+		"2009", "2010", "2011", "2012", "monday", "tuesday", "wednesday", "thirsday", "friday", "saturday", "sunday",
+		"mondays", "tuesdays", "wednesdays", "thirsdays", "fridays", "saturdays", "sundays", "men" // ?
+	});
+
+	private BingQueryRunner webSearch = new BingQueryRunner();
+
+	private StringDistanceMeasurer stringDistanceMeasurer = new StringDistanceMeasurer();
+
+
+	public SimilarityAccessorBase()
+	{
+	}
+
+
+	public void init()
+	{
+		namesBothSides = getWordsThatShouldBeOnBothSidesEvents();
+	}
+
+	protected List<String> removeDollarWordAndNonAlphaFromList(List<String> list)
+	{
+		List<String> result = new ArrayList<String>();
+		Pattern p = Pattern.compile("^\\$(\\d{1,3}(\\,\\d{3})*|(\\d+))(\\.\\d{2})?$");
+		for (String w : list)
+		{
+			if (!(p.matcher(w).find()) && StringUtils.isAlphanumeric(w) && (w.length() >= 3 || !StringUtils.isAlpha(w)))
+				result.add(w);
+		}
+		return result;
+	}
+
+
+	public List<String> getWordsThatShouldBeOnBothSidesEvents()
+	{
+/*
+		names.addAll(Arrays.asList(new String[] { "woman", "man", "women", "men", "womans", "mans", "womens", "mens",
+			"boy", "girl", "boys", "girls", "men's", "women's", "woman's", "ice", // for disney
+			"flight", "intermediate", "advanced", "beginner",
+			// "tour", TODO special consideration
+			"helicopter", "sexual", "junior", "jr" }));
+			*/
+		return null;
+
+	}
+
+	protected Boolean applySemanticNameSimilarityRule(Object es1,
+		Object es2)
+	{
+		
+		//TODO check attributes of objects
+		/*
+		if (!(es1.getVenueName().endsWith(es2.getVenueName()) || es2.getVenueName().endsWith(es1.getVenueName())))
+			return false;
+		if (Math.abs(es1.getStarttime().getTime() - es2.getStarttime().getTime()) > 100000)
+			return false;
+			*/
+
+		return true;
+
+	}
+
+	// this rule extract "OF" part and treats it as a whole expression
+	protected void applySubPhraseExtractionRule(List<String> name1Tokens, List<String> name2Tokens)
+	{
+		if (name1Tokens.indexOf("of") > 0 && name2Tokens.indexOf("of") > 0)
+		{
+			name1Tokens = extractMainNounPhrase(name1Tokens);
+			name2Tokens = extractMainNounPhrase(name2Tokens);
+		}
+	}
+
+	private Boolean attemptShortTitlesSimilarityInWebSpace(String name1, String name2)
+	{
+
+		// first delimeter processing
+		String name1v = name1.replace("'", "").replace("-", " ");
+		String name2v = name2.replace("'", "").replace("-", " ");
+		String name1vv = name1.replace("'", "");
+		String name2vv = name2.replace("'", "");
+		String name1vvv = name1.replace("-", " ");
+		String name2vvv = name2.replace("-", " ");
+
+		if (name1.startsWith(name2) || name1vv.startsWith(name2) || name1.startsWith(name2v)
+			|| name1.startsWith(name2vv) || name1.startsWith(name2vvv) || name1v.startsWith(name2v)
+			|| name1v.startsWith(name2vv) || name2.startsWith(name1) || name2vv.startsWith(name1)
+			|| name2.startsWith(name1v) || name2vvv.startsWith(name1vv) || name2.startsWith(name1vvv)
+			|| name2v.startsWith(name1v) || name2v.startsWith(name1vv) || name1.endsWith(name2)
+			|| name1vv.endsWith(name2) || name1.endsWith(name2v) || name1.endsWith(name2vv) || name1.endsWith(name2vvv)
+			|| name1v.endsWith(name2v) || name1v.endsWith(name2vv) || name2.endsWith(name1) || name2vv.endsWith(name1)
+			|| name2.endsWith(name1v) || name1vvv.endsWith(name2vv) || name2.endsWith(name1vvv)
+			|| name2v.endsWith(name1v) || name2v.endsWith(name1vv))
+		{
+			LOG.info("Found fuzzy substring of name1 and name2");
+			return true;
+		}
+		if (name1.length() > 12 && name2.length() > 12)
+			return false;
+
+		return areNamesSemanticallyCloseInWebSearchSpace(name1, name2, 0.8f, false).isDecision();
+
+	}
+
+	public Boolean applyBothSidesRuleEvent(String name1, String name2)
+	{
+		List<String> name1Tokens = TextProcessor.fastTokenize(name1.toLowerCase(), false);
+		List<String> name2Tokens = TextProcessor.fastTokenize(name2.toLowerCase(), false);
+		// get unique names
+		List<String> name1TokensC = new ArrayList<String>(name1Tokens), name2TokensC = new ArrayList<String>(
+			name2Tokens);
+		;
+		name1TokensC.removeAll(name2Tokens);
+		name2TokensC.removeAll(name1Tokens);
+		// get all unique names
+		name1TokensC.addAll(name2TokensC);
+
+		name1TokensC.retainAll(namesBothSides);
+		name1Tokens.retainAll(name2Tokens);
+
+		if ((name1TokensC.size() > 0 && name1Tokens.size() < 3) || (name1TokensC.size() > 1 && name1Tokens.size() < 5))
+		{ // 'mens == men; case !(name1TokensC.size()==2 && (name1TokensC.get(0).indexOf(name1TokensC.get(1))>-1 ||
+			// name1TokensC.get(1).indexOf(name1TokensC.get(0))>-1 ))){
+			LOG.info("Found required common word present on one side and not on the other: " + name1TokensC.toString()
+				+ " and less than 3 keywords overlap (or >1 common words and less than 5 overl");
+			return false;
+		}
+		else
+			return true;
+	}
+
+	protected List<String> tokenizeAndStem(String input)
+	{
+
+		List<String> results = new ArrayList<String>();
+		List<String> toks = TextProcessor.fastTokenize(input.toLowerCase(), false);
+		for (String word : toks)
+		{
+			try
+			{
+				if (word.equals("theatre"))
+					word = "theater";
+				results.add(word);
+			}
+			catch (Exception e)
+			{
+				results.add(word);
+			}
+		}
+		return results;
+	}
+
+	protected List<String> stemList(List<String> toks)
+	{
+
+		List<String> results = new ArrayList<String>();
+		for (String word : toks)
+		{
+			try
+			{
+				if (word.equals("theatre"))
+					word = "theater";
+				results.add(word);
+			}
+			catch (Exception e)
+			{
+				results.add(word);
+			}
+		}
+		return results;
+	}
+
+	public List<String> removeVenuePart(ArrayList<String> toks)
+	{
+		List<String> results = new ArrayList<String>();
+		boolean bVenuePart = false;
+		for (String word : toks)
+		{
+			// beginning of venue part
+			if (word.equals("at") || word.equals("@"))
+				bVenuePart = true;
+			// end of venue part
+			if (!StringUtils.isAlphanumeric(word) || word.startsWith("<punc"))
+				bVenuePart = false;
+
+			if (!bVenuePart && !word.startsWith("<punc"))
+				results.add(word);
+
+		}
+		return results;
+	}
+
+	protected boolean isCapitalized(String lookup)
+	{
+		String[] titleWords = lookup.split(" ");
+		int count = 0;
+		for (String word : titleWords)
+		{
+			if (word.length() < 2) // '-', '|', ':'
+				break;
+
+			if (word.equals(word.toLowerCase()) && (!Arrays.asList(englishPrepositions).contains(word))
+				&& word.length() > 3 && StringUtils.isAlphanumeric(word))
+				continue; // was return false;
+			if (count > 3)
+				break;
+			count++;
+		}
+		return true;
+	}
+
+	protected List<String> extractMainNounPhrase(List<String> name1Tokens)
+	{
+		List<String> results = new ArrayList<String>();
+		int ofPos = name1Tokens.indexOf("of");
+		List<String> ofList = name1Tokens.subList(ofPos + 1, name1Tokens.size() - 1);
+		// now iterate till next preposition towards the end of noun phrase
+		for (String preposCand : ofList)
+		{
+			if (Arrays.asList(englishPrepositions).contains(preposCand))
+				break;
+			results.add(preposCand);
+		}
+		return results;
+
+	}
+
+	public boolean verifyEventAttributesPost(List<String> name1Tokens, List<String> name2Tokens)
+	{
+		String[] attributeNamesPost = { "age", "ages", "game", "games", "grade", "grades", "level", "levels", "vs",
+			"vs.", "versus", "pottery", "competition", "contest", "skill", "skills", "day", "only", "basic", "class",
+			"completed",
+			// "tour", ?
+			"advanced", "beginner", "intermediate", "flight", "workshop", "latin", "adobe", "ballet", "dinner",
+			"breakfast", "lunch", "summer", // "canyon"
+			"tfestival", "festival", "mfestival" };
+		try
+		{
+			for (String attr : attributeNamesPost)
+			{
+
+				int agePos1 = name1Tokens.indexOf(attr);
+				int agePos2 = name2Tokens.indexOf(attr);
+				if (agePos1 > -1 && agePos2 > -1 && agePos1 < name1Tokens.size() - 1
+					&& agePos2 < name2Tokens.size() - 1)
+				{
+					double dist = LevensteinDistanceFinder.levensteinDistance(name1Tokens.get(agePos1 + 1),
+						name2Tokens.get(agePos2 + 1), 1, 10, 1, 10);
+					if (!name1Tokens.get(agePos1 + 1).equalsIgnoreCase(name2Tokens.get(agePos2 + 1))
+						&& (dist > 2.99 || name1Tokens.get(agePos1 + 1).length() < 4))
+					{
+						LOG.info("Found disagreement in the attrib value for " + attr + " value = "
+							+ name1Tokens.get(agePos1 + 1) + " <=> " + name2Tokens.get(agePos2 + 1));
+						return false;
+					}
+				}
+			}
+		}
+		catch (Exception e)
+		{
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		}
+		return true;
+	}
+
+	public boolean verifyEventAttributesPre(List<String> name1Tokens, List<String> name2Tokens)
+	{
+
+		String[] attributeNamesPre = { "hour", "vs", "vs.", "versus", "pottery", "program", "day", "only",
+			// dance styles followed by a param
+			"swing", "rumba", "samba", "doble",
+			"violence", //
+			// "level",
+			"class", "classes", "kid", "kids", "test", "west", "summer_camp", "session", "tfestival", "festival",
+			"mfestival" };
+		try
+		{
+			for (String attr : attributeNamesPre)
+			{
+				int agePos1 = name1Tokens.indexOf(attr);
+				int agePos2 = name2Tokens.indexOf(attr);
+				if (agePos1 > 0 && agePos2 > 0)
+				{ // not the first word is attr name
+					if (!name1Tokens.get(agePos1 - 1).equalsIgnoreCase(name2Tokens.get(agePos2 - 1))
+						&& (agePos1 < 2 || !name1Tokens.get(agePos1 - 2).equalsIgnoreCase(name2Tokens.get(agePos2 - 1)))
+						&&
+						// ((agePos1<2 && agePos2 <2) || !name1Tokens.get(agePos1 -
+						// 2).equalsIgnoreCase(name2Tokens.get(agePos2 - 2 ))) &&
+						(agePos2 < 2 || !name1Tokens.get(agePos1 - 1).equalsIgnoreCase(name2Tokens.get(agePos2 - 2)))
+
+					)
+					{
+						LOG.info("Found disagreement in the attrib value for " + attr + " value = "
+							+ name1Tokens.get(agePos1 - 1) + " and " + name2Tokens.get(agePos2 - 1));
+						return false;
+					}
+				}
+			}
+		}
+		catch (Exception e)
+		{
+			e.printStackTrace();
+		}
+		return true;
+	}
+
+	protected boolean bDifferentGroupOneSubnameOfAnother(String name1, String name2)
+	{
+		// first check a special case that both name1 and name2 are DIFFERENT groups at last.fm
+		Map<String, Integer> map1 = null; //LastFM_APIManager.extractTagsForArtist(name1);
+		Map<String, Integer> map2 = null; //LastFM_APIManager.extractTagsForArtist(name2);
+		if (map1 != null && map2 != null && map1.size() > 0 && map2.size() > 0)
+			map1.entrySet().removeAll(map2.entrySet());
+		if (map1.size() > 0) // same or subset of tags => different groups
+			return true;
+
+		return false;
+	}
+
+	public boolean applyBothSidesRule(String name1, String name2)
+	{
+		List<String> name1Tokens = TextProcessor.fastTokenize(name1.toLowerCase(), false);
+		List<String> name2Tokens = TextProcessor.fastTokenize(name2.toLowerCase(), false);
+		// get unique names
+		List<String> name1TokensC = new ArrayList<String>(name1Tokens), name2TokensC = new ArrayList<String>(
+			name2Tokens);
+		;
+		name1TokensC.removeAll(name2Tokens);
+		name2TokensC.removeAll(name1Tokens);
+		// get all unique names
+		name1TokensC.addAll(name2TokensC);
+
+		name1TokensC.retainAll(namesBothSides);
+		if (name1TokensC.size() > 0)
+			return false;
+		else
+			return true;
+	}
+
+	private boolean succeededMenWomenSportsRule(String name1, String name2)
+	{
+		List<String> name1Tokens = TextProcessor.fastTokenize(name1.toLowerCase(), false);
+		List<String> name2Tokens = TextProcessor.fastTokenize(name2.toLowerCase(), false);
+		if (name1Tokens.contains("men") || name2Tokens.contains("men") || name1Tokens.contains("women")
+			|| name2Tokens.contains("women") || name1Tokens.contains("disney") || name2Tokens.contains("disney"))
+		{ // all words should be the
+			// same
+			name1Tokens.removeAll(name2Tokens);
+			name1Tokens.removeAll(Arrays.asList(englishPrepositions));
+			name1Tokens.removeAll(Arrays.asList(commonWordsInEventTitles));
+			if (name1Tokens.size() < 1)
+				return true;
+
+			return false;
+		}
+		else
+			return true;
+
+	}
+
+	private boolean succeededSpecialGroupsSymphoniesRule(String name1, String name2)
+	{
+		List<String> name1Tokens = TextProcessor.fastTokenize(name1.toLowerCase(), false);
+		List<String> name2Tokens = TextProcessor.fastTokenize(name2.toLowerCase(), false);
+		if (name1Tokens.contains("orchestra") || name2Tokens.contains("symphony") || name2Tokens.contains("orchestra")
+			|| name1Tokens.contains("symphony") || name2Tokens.contains("band") || name1Tokens.contains("band")
+			|| name2Tokens.contains("trio") || name1Tokens.contains("trio") || name1Tokens.contains("soleil")
+			|| name2Tokens.contains("soleil") || name1Tokens.contains("disney") || name2Tokens.contains("disney")
+			|| name1Tokens.contains("lang") || name2Tokens.contains("lang")) // special group 'lang lang'
+		{ // all words should be the
+			// same
+			List<String> name1TokensClone = new ArrayList<String>(name1Tokens);
+			name1Tokens.removeAll(name2Tokens);
+			name2Tokens.removeAll(name1TokensClone);
+			name1Tokens.addAll(name2Tokens);
+			name1Tokens.removeAll(Arrays.asList(this.englishPrepositions));
+			// name1Tokens.removeAll(Arrays.asList(this.commonWordsInEventTitles));
+			if (name1Tokens.size() < 1)
+				return true;
+
+			return false;
+		}
+		else
+			return true;
+
+	}
+
+	public int getAttemptedNameMerge(String name1, String name2)
+	{
+		name1 = name1.replaceAll("[a-z][A-Z]", "$0&$0").replaceAll(".&.", " ");
+		; // suspected word merge if higher case is in the middle of word
+		name2 = name2.replaceAll("[a-z][A-Z]", "$0&$0").replaceAll(".&.", " ");
+
+		name1 = name1.toLowerCase();
+		name2 = name2.toLowerCase();
+		if (name1.equals(name2) || name1.startsWith(name2) || name2.startsWith(name1) || name1.endsWith(name2)
+			|| name1.endsWith(name2) || name1.indexOf(name2) > -1 || name1.indexOf(name2) > -1) // ??
+			return 2;
+		String name2r = name2.replace(" ", "");
+		if (name1.equals(name2r) || name1.startsWith(name2r) || name1.startsWith(name2r) || name1.endsWith(name2r)
+			|| name1.endsWith(name2r))
+			return 1;
+		String name1r = name1.replace(" ", "");
+		if (name1r.equals(name2r) || name1r.startsWith(name2r) || name1r.startsWith(name2) || name1r.endsWith(name2r)
+			|| name1r.endsWith(name2r) || name2r.equals(name1r) || name2r.startsWith(name1r)
+			|| name2r.startsWith(name1) || name2r.endsWith(name1r) || name2r.endsWith(name2)
+
+		)
+			return 1;
+
+		if (stringDistanceMeasurer.measureStringDistance(name1, name2) > 0.95)
+			return 2;
+		if (stringDistanceMeasurer.measureStringDistance(name1, name2) > 0.70)
+			return 1;
+		return 0;
+	}
+
+	private String normalizeGenderAndOtherAttributes(String name1)
+	{
+		name1 = Utils.convertToASCII(name1.replace("/", " ").replace("w/", "with ")).replace('!', ' ').toLowerCase();
+
+		name1 = name1.replace("woman", "women").replace("womans", "women").replace("womens", "women")
+			.replace("women's", "women").replace("woman's", "women");
+		name1 = name1.replace(" man ", " men ").replace(" mans ", " men ").replace(" men's ", " men ")
+			.replace(" man's ", " men ").replace(" mens ", " men ").replace("summer camp", "summer_camp")
+			.replace("gaea theatre festival", "tfestival"); // need regexp for this
+		return name1;
+	}
+
+	/*
+	 * Main semantic similarity function which applies boundary cases rule and focus on web mining rule The main
+	 * criteria for a commonality between titles: to form an entity, searchable on the web
+	 */
+	public DedupResult areNamesSemanticallyCloseWebMineCommonPart(String name1, String name2, String venue)
+	{
+		// normalize gender
+		name1 = normalizeGenderAndOtherAttributes(name1);
+		name2 = normalizeGenderAndOtherAttributes(name2);
+
+		Boolean bShortTitlesSimilarInWebSpace = attemptShortTitlesSimilarityInWebSpace(name1, name2);
+		if (bShortTitlesSimilarInWebSpace)
+			return new DedupResult("Accepted as short title by web mining", 2, true);
+
+		StringBuffer reason = new StringBuffer();
+		List<String> venueToks = removeVenuePart(TextProcessor.fastTokenize(venue.toLowerCase(), false));
+
+		LOG.info("\nComputing similarity between name = '" + name1 + "' and name = '" + name2 + "'");
+		// convert titles into token lists
+		List<String> name1Tokens = removeVenuePart(TextProcessor.fastTokenize(name1.toLowerCase(), true));
+		List<String> name2Tokens = removeVenuePart(TextProcessor.fastTokenize(name2.toLowerCase(), true));
+		// applySubPhraseExtractionRule()
+		Boolean bSameAttrib = verifyEventAttributesPost(name1Tokens, name2Tokens)
+			&& verifyEventAttributesPre(name1Tokens, name2Tokens);
+		if (!bSameAttrib)
+		{
+			LOG.info("similar events but different attributes");
+			return new DedupResult("similar events but different attributes", 0, false);
+		}
+
+		boolean bothSodesSuccess = applyBothSidesRuleEvent(name1, name2);
+		if (!bothSodesSuccess)
+		{
+			return new DedupResult("Failed common words test for sports", 0, false);
+		}
+
+		float dist = (float) LevensteinDistanceFinder.levensteinDistance(name1, name2, 1, 10, 1, 10);
+		if (dist < 5.1)
+		{
+			LOG.info("Found low LevensteinDistance for name1 and name2");
+			return new DedupResult("Found low LevensteinDistance", 2, true);
+		}
+
+		int nameMergeScore = getAttemptedNameMerge(name1, name2);
+		if (nameMergeScore > 0)
+		{
+			LOG.info("Found low NameMerge Distance for name1 and name2");
+			return new DedupResult("Found low  NameMerge Distance", 2, true);
+		}
+
+		// todo take into account order
+		// form common sub-list of tokens
+		name1Tokens.retainAll(name2Tokens);
+		name1Tokens.removeAll(venueToks);
+
+		name1Tokens.removeAll(commonWordsInEventTitles);
+		name1Tokens.removeAll(Arrays.asList(englishPrepositions));
+		name1Tokens = removeDollarWordAndNonAlphaFromList(name1Tokens);
+		// todo : to use full string measure
+		// boundary case: too many words => just do counts
+		float commonPortion = (float) name1Tokens.size() / (float) name2Tokens.size();
+		if (commonPortion > 0.8 || name1Tokens.size() >= 4)
+		{ // after typical
+			// title words
+			// are revomed 4
+			// looks OK
+			LOG.info("Accepted since substantial common part");
+			return new DedupResult("Accepted since substantial common part", Math.max((int) (commonPortion * 5.0), 2),
+				true);
+		}
+		// boundary case: no overlap
+		if (name1Tokens.size() < 1)
+		{
+			LOG.info("Rejected since nothing in common");
+			return new DedupResult("Rejected since nothing in common", 0, false);
+		}
+		// get from list of tokens back to words to get search expression
+		String entityExpression = name1Tokens.toString().replace('[', ' ').replace(']', ' ').replace(',', ' ')
+			.replace("  ", " ").trim();
+		/*
+		 * // now try name merge reduced strings String entityExpression1 = name1TokensC.toString().replace('[',
+		 * ' ').replace(']', ' ').replace(',', ' ') .replace("  ", " ").trim(); String entityExpression2 =
+		 * name2Tokens.toString().replace('[', ' ').replace(']', ' ').replace(',', ' ') .replace("  ", " ").trim();
+		 * 
+		 * nameMergeScore = getAttemptedNameMerge(entityExpression1, entityExpression2); if (nameMergeScore>0){
+		 * LOG.info("Found low NameMerge Distance for REDUCED name1 and name2"); return new
+		 * DedupResult("Found low  NameMerge Distance REDUCED", 2, true);
+		 * 
+		 * }
+		 */
+
+		// Before doing web mining, make sure overlap between titles is NOT a
+		// set of common english words (use the vocabulary)
+		// if all words are common, then NOT an entity
+		if (name1Tokens.size() < 2)
+		{
+			boolean bCommonEnglishWord = false;
+			for (String word : name1Tokens)
+			{
+	//			if (stopList.isCommonWord(word) /*&& mostFrequent1000Words.isMostFrequent1000Word(word)*/)
+	//				bCommonEnglishWord = true;
+			}
+
+			if (bCommonEnglishWord)
+			{
+				LOG.info("Rejected common entity: common word = " + entityExpression);
+				return new DedupResult("Rejected since common entity is common English word = " + entityExpression, 0,
+					false);
+			}
+		}
+		// accept common expression
+		LOG.info("Formed common entity = " + entityExpression);
+		reason.append("Formed common entity = " + entityExpression + "\n");
+		// now go to the web / bing api with this common expression
+		List<HitBase> searchResult = webSearch.runSearch(entityExpression);
+		float entityScore = 0f;
+		if (searchResult != null)
+		{
+			int count = 0;
+			for (HitBase item : searchResult)
+			{
+				String lookup = item.getTitle();
+				LOG.info("Bing hit title = '" + lookup + "'");
+				reason.append("Bing hit title = '" + lookup + "'\n");
+				if (count > 4)
+					break;
+				count++;
+				// if occurrence is not capitalized then rejected, do not take
+				// into account in score
+				if (!isCapitalized(lookup))
+				{
+					LOG.info("Rejected hit title since not capitalized");
+					reason.append("Rejected hit title since not capitalized\n");
+					continue;
+				}
+
+				/*
+				 * if (lookup.indexOf('-')>0 ){ lookup = lookup.split("-")[0]; }
+				 */
+				// now compute overlap between what found on the web for hit's
+				// title and the common expression between events
+				List<String> lookupTokens = tokenizeAndStem(lookup);
+				lookupTokens.retainAll(stemList(name1Tokens));
+				if (lookupTokens.size() >= name1Tokens.size())
+					// increment score if found hit title is acceptable
+					entityScore += 1.0;
+				else
+				{
+					LOG.info("Found hit title " + lookupTokens + " does not cover comonality expr = " + name1Tokens);
+					entityScore += 0.25;
+
+				}
+
+			}
+		}
+		return new DedupResult(reason.toString(), (int) entityScore, entityScore > 1.0);
+	}
+
+	public DedupResult areNamesSemanticallyCloseInWebSearchSpace(String name1, String name2, Float thresh, boolean bStem)
+	{
+
+		if (thresh == null || thresh == 0f)
+		{
+			thresh = 0.8f;
+		}
+
+		// normalize gender
+		name1 = normalizeGenderAndOtherAttributes(name1);
+		name2 = normalizeGenderAndOtherAttributes(name2);
+
+		StringBuffer reason = new StringBuffer();
+
+		boolean bSportsOrOrchestra = !succeededMenWomenSportsRule(name1, name2);
+		if (bSportsOrOrchestra)
+			return new DedupResult("Sports rule: different teams or teams of different venues", 0, false);
+
+		bSportsOrOrchestra = !succeededSpecialGroupsSymphoniesRule(name1, name2);
+		if (bSportsOrOrchestra)
+			return new DedupResult("SpecialGroupsSymphoniesRule: different circus/band", 0, false);
+
+		LOG.info("\nComputing similarity between name = '" + name1 + "' and name = '" + name2 + "'");
+
+		List<String> name1Tokens = TextProcessor.fastTokenize(name1.toLowerCase(), true);
+		List<String> name2Tokens = TextProcessor.fastTokenize(name2.toLowerCase(), true);
+		Boolean bSameAttrib = verifyEventAttributesPost(name1Tokens, name2Tokens)
+			&& verifyEventAttributesPre(name1Tokens, name2Tokens);
+		if (!bSameAttrib)
+		{
+			LOG.info("similar events but different attributes");
+			return new DedupResult("similar events but different attributes", 0, false);
+		}
+
+		List<HitBase> searchResult1 = webSearch.runSearch(name1);
+		List<HitBase> searchResult2 = webSearch.runSearch(name2);
+		int score = 0;
+		if (searchResult1 != null && searchResult2 != null)
+		{
+			for (HitBase item1 : searchResult1)
+			{
+				if (item1.getUrl().indexOf("myspace") > -1 || item1.getUrl().indexOf("wiki") > -1)
+					continue;
+				for (HitBase item2 : searchResult2)
+				{
+					String lookup1 = item1.getTitle().replace("Facebook", "").replace("LinkedIn", "")
+						.replace("MySpace", "");
+					String lookup2 = item2.getTitle().replace("Facebook", "").replace("LinkedIn", "")
+						.replace("MySpace", "");
+					double d = 0;
+					if (bStem)
+						d = stringDistanceMeasurer.measureStringDistance(lookup1, lookup2);
+					else
+						d = stringDistanceMeasurer.measureStringDistanceNoStemming(lookup1, lookup2);
+					if (d > thresh) // 0.8)
+					{
+
+						reason.append("Found common search result title for group names '" + lookup1 + " < > "
+							+ lookup2 + " sim = " + d + "\n");
+						LOG.info(("Found common search result title for group names '" + lookup1 + " < > " + lookup2
+							+ " sim = " + d));
+						score++;
+					}
+
+				}
+			}
+		}
+
+		Boolean bothSidesSuccess = applyBothSidesRule(name1, name2);
+		if (!bothSidesSuccess)
+		{
+			score = 1;
+			reason.append("Failed common words test for sports");
+		}
+		if (score > 0)
+		{
+			Boolean bDifferentGroup = bDifferentGroupOneSubnameOfAnother(name1, name2);
+			if (bDifferentGroup)
+			{
+				score = 1;
+				reason.append("Failed common words test for sports");
+			}
+		}
+		return new DedupResult(reason.toString(), score, score > 1);
+	}
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/doc_classifier.zip
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/doc_classifier.zip b/opennlp-similarity/src/main/java/opennlp/tools/doc_classifier.zip
new file mode 100644
index 0000000..2227504
Binary files /dev/null and b/opennlp-similarity/src/main/java/opennlp/tools/doc_classifier.zip differ

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/doc_classifier/ClassifierTrainingSetIndexer.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/doc_classifier/ClassifierTrainingSetIndexer.java b/opennlp-similarity/src/main/java/opennlp/tools/doc_classifier/ClassifierTrainingSetIndexer.java
new file mode 100644
index 0000000..ad851e3
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/doc_classifier/ClassifierTrainingSetIndexer.java
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.doc_classifier;
+
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+import opennlp.tools.jsmlearning.ProfileReaderWriter;
+
+import org.apache.lucene.analysis.standard.StandardAnalyzer;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.document.Field;
+import org.apache.lucene.document.StringField;
+import org.apache.lucene.document.TextField;
+import org.apache.lucene.index.IndexWriter;
+import org.apache.lucene.index.IndexWriterConfig;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.store.FSDirectory;
+import org.apache.lucene.util.Version;
+import org.apache.tika.Tika;
+
+public class ClassifierTrainingSetIndexer {
+	public static String resourceDir = new File(".").getAbsolutePath().replace("/.", "") + "/src/main/resources";
+    public static String INDEX_PATH = "/classif",
+            CLASSIF_TRAINING_CORPUS_PATH = "/training_corpus";
+    protected ArrayList<File> queue = new ArrayList<File>();
+    Tika tika = new Tika();
+
+    IndexWriter indexWriter = null;
+    protected static String[] domains =  new String[] { "legal", "health",
+   	 "computing", "engineering", "business" };
+	private String absolutePathTrainingSet=null;
+
+    public ClassifierTrainingSetIndexer() {
+
+        try {
+            initIndexWriter(resourceDir);
+        } catch (Exception e) {
+            e.printStackTrace();
+        }
+    } 
+
+    public ClassifierTrainingSetIndexer(String absolutePathTrainingSet) {
+    	this.absolutePathTrainingSet = absolutePathTrainingSet;
+        try {
+            initIndexWriter(resourceDir);
+        } catch (Exception e) {
+            e.printStackTrace();
+        }
+    }
+
+    public void indexTrainingSet() {
+        
+        try {
+        	if (absolutePathTrainingSet==null)
+            indexFileOrDirectory(resourceDir
+                    + CLASSIF_TRAINING_CORPUS_PATH);
+        	else
+        		 indexFileOrDirectory(
+                         this.absolutePathTrainingSet);
+        		
+        } catch (IOException e1) {
+            e1.printStackTrace();
+        }
+        try {
+            indexWriter.commit();
+        } catch (IOException e) {
+            e.printStackTrace();
+        }
+    }
+/*
+    private void indexTrainingSample(String text, String flag, int id)
+            throws IOException {
+
+        Document doc = new Document();
+        doc.add(new StringField("id", new Integer(id).toString(),
+                Field.Store.YES));
+        doc.add(new TextField("text", text.toLowerCase(), Field.Store.YES));
+        doc.add(new StringField("class", flag.toLowerCase(), Field.Store.YES));
+        indexWriter.addDocument(doc);
+
+    }
+*/
+    private void addFiles(File file) {
+
+        if (!file.exists()) {
+            System.out.println(file + " does not exist.");
+        }
+        if (file.isDirectory()) {
+            for (File f : file.listFiles()) {
+                if (f.getName().startsWith("."))
+                    continue;
+                addFiles(f);
+                System.out.println(f.getName());
+            }
+        } else {
+            queue.add(file);
+
+        }
+    }
+
+    // index last folder name, before filename itself
+
+    public void indexFileOrDirectory(String fileName) throws IOException {
+        addFiles(new File(fileName));
+
+        List<File> files = new ArrayList<File>(queue);
+        for (File f : files) {
+            if (!f.getName().endsWith(".xml")) {
+
+                try {
+                    Document doc = new Document();
+
+                    String name = f.getPath();
+                    String className = null;
+                    for (String d : domains) {
+                        if (name.indexOf(d) > -1) {
+                            className = d;
+                            break;
+                        }
+                    }
+
+                    try {
+                        doc.add(new TextField("text", tika.parse(f)));
+                    } catch (Exception e1) {
+                        e1.printStackTrace();
+                    }
+
+                    doc.add(new StringField("path", f.getPath(),
+                            Field.Store.YES));
+                    doc.add(new StringField("class", className, Field.Store.YES));
+                    try {
+
+                        indexWriter.addDocument(doc);
+
+                    } catch (Exception e) {
+                        e.printStackTrace();
+                        System.out.println("Could not add: " + f);
+                    }
+                } catch (Exception ee) {
+                    ee.printStackTrace();
+                }
+            } else { // for xml files
+                try {
+                    Document doc = new Document();
+
+                    String name = new String(f.getPath());
+                    String[] nparts = name.split("/");
+                    int len = nparts.length;
+                    name = nparts[len - 2];
+
+                    FileReader fr = new FileReader(f);
+                    doc.add(new TextField("text", fr));
+
+                    doc.add(new StringField("path", f.getPath(),
+                            Field.Store.YES));
+                    doc.add(new StringField("class", name, Field.Store.YES));
+                    try {
+
+                        indexWriter.addDocument(doc);
+
+                    } catch (Exception e) {
+                        e.printStackTrace();
+                        System.out.println("Could not add: " + f);
+                    } finally {
+                        fr.close();
+                    }
+                } catch (Exception ee) {
+                    ee.printStackTrace();
+                }
+            }
+
+            queue.clear();
+        }
+    }
+
+    public static String getIndexDir() {
+        try {
+            return new File(".").getCanonicalPath() + INDEX_PATH;
+        } catch (IOException e) {
+            // TODO Auto-generated catch block
+            e.printStackTrace();
+            return null;
+        }
+    }
+
+    private void initIndexWriter(String dir) throws Exception {
+
+        Directory indexDir = null;
+
+        try {
+            indexDir = FSDirectory.open(new File(dir + INDEX_PATH));
+        } catch (IOException e) {
+            // TODO Auto-generated catch block
+            e.printStackTrace();
+        }
+
+        Version luceneVersion = Version.LUCENE_46;
+        IndexWriterConfig luceneConfig = new IndexWriterConfig(luceneVersion,
+                new StandardAnalyzer(luceneVersion));
+        luceneConfig.setOpenMode(IndexWriterConfig.OpenMode.CREATE);
+
+        indexWriter = new IndexWriter(indexDir, luceneConfig);
+
+    }
+
+    void close() {
+        try {
+            indexWriter.commit();
+            indexWriter.close();
+        } catch (IOException e) {
+            // TODO Auto-generated catch block
+            e.printStackTrace();
+        }
+    }
+    
+    public static String getCategoryFromFilePath(String path){
+    	String className = null;
+        for (String d : domains) {
+            if (path.indexOf("/"+d+"/") > -1) {
+                className = d;
+                break;
+            }
+        }
+        return className;
+    }
+
+    public static void main(String[] args) {
+    	ClassifierTrainingSetIndexer indexer = null;
+    	if (args!=null && args.length==1){
+	    	String relativeDirWithTrainingCorpus = args[0];
+	    	// expect corpus relative to 'resource' directory, such as 'training_corpus'
+	    	if (!relativeDirWithTrainingCorpus.startsWith("/"))
+	    		relativeDirWithTrainingCorpus = "/"+relativeDirWithTrainingCorpus;
+	        indexer = new ClassifierTrainingSetIndexer(relativeDirWithTrainingCorpus);
+    	} else {
+    		// expect corpus in the default location, "/training_corpus" in the resource directory
+    		indexer = new ClassifierTrainingSetIndexer();
+    	}
+        try {
+            indexer.indexTrainingSet();
+        } catch (Exception e) {
+            e.printStackTrace();
+        }
+        indexer.close();
+    }
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/doc_classifier/DocClassifier.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/doc_classifier/DocClassifier.java b/opennlp-similarity/src/main/java/opennlp/tools/doc_classifier/DocClassifier.java
new file mode 100644
index 0000000..b4abbb9
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/doc_classifier/DocClassifier.java
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.doc_classifier;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Scanner;
+
+import opennlp.tools.similarity.apps.utils.CountItemsList;
+import opennlp.tools.similarity.apps.utils.ValueSortMap;
+import opennlp.tools.textsimilarity.TextProcessor;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.lucene.analysis.Analyzer;
+import org.apache.lucene.analysis.standard.StandardAnalyzer;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.index.DirectoryReader;
+import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.queryparser.classic.ParseException;
+import org.apache.lucene.queryparser.classic.QueryParser;
+import org.apache.lucene.search.IndexSearcher;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.ScoreDoc;
+import org.apache.lucene.search.TopDocs;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.store.FSDirectory;
+import org.apache.lucene.util.Version;
+import org.json.JSONObject;
+
+public class DocClassifier {
+	public static final String DOC_CLASSIFIER_KEY = "doc_class";
+	public static String resourceDir = null;
+	public static final Log logger = LogFactory.getLog(DocClassifier.class);
+	private Map<String, Float> scoredClasses = new HashMap<String, Float>();
+	
+
+	public static Float MIN_TOTAL_SCORE_FOR_CATEGORY = 0.3f; //3.0f;
+	protected static IndexReader indexReader = null;
+	protected static IndexSearcher indexSearcher = null;
+	// resource directory plus the index folder
+	private static final String INDEX_PATH = resourceDir
+			+ ClassifierTrainingSetIndexer.INDEX_PATH;
+
+	// http://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm
+	private static final int MAX_DOCS_TO_USE_FOR_CLASSIFY = 10, // 10 similar
+			// docs for
+			// nearest
+			// neighbor
+			// settings
+
+			MAX_CATEG_RESULTS = 2;
+	private static final float BEST_TO_NEX_BEST_RATIO = 2.0f;
+	// to accumulate classif results
+	private CountItemsList<String> localCats = new CountItemsList<String>();
+	private int MAX_TOKENS_TO_FORM = 30;
+	private String CAT_COMPUTING = "computing";
+	public static final String DOC_CLASSIFIER_MAP = "doc_classifier_map";
+	private static final int MIN_SENTENCE_LENGTH_TO_CATEGORIZE = 60; // if
+	// sentence
+	// is
+	// shorter,
+	// should
+	// not
+	// be
+	// used
+	// for
+	// classification
+	private static final int MIN_CHARS_IN_QUERY = 30; // if combination of
+	// keywords is shorter,
+	// should not be used
+	// for classification
+
+	// these are categories from the index
+	public static final String[] categories = new String[] { "legal", "health",
+		"finance", "computing", "engineering", "business" };
+
+	static {
+		synchronized (DocClassifier.class) {
+			Directory indexDirectory = null;
+
+			try {
+				indexDirectory = FSDirectory.open(new File(INDEX_PATH));
+			} catch (IOException e2) {
+				logger.error("problem opening index " + e2);
+			}
+			try {
+				indexReader = DirectoryReader.open(indexDirectory);
+				indexSearcher = new IndexSearcher(indexReader);
+			} catch (IOException e2) {
+				logger.error("problem reading index \n" + e2);
+			}
+		}
+	}
+
+	public DocClassifier(String inputFilename, JSONObject inputJSON) {
+		scoredClasses = new HashMap<String, Float>();
+	}
+
+	/* returns the class name for a sentence */
+	private List<String> classifySentence(String queryStr) {
+
+		List<String> results = new ArrayList<String>();
+		// too short of a query
+		if (queryStr.length() < MIN_CHARS_IN_QUERY) {
+			return results;
+		}
+
+		Analyzer std = new StandardAnalyzer(Version.LUCENE_46);
+		QueryParser parser = new QueryParser(Version.LUCENE_46, "text", std);
+		parser.setDefaultOperator(QueryParser.Operator.OR);
+		Query query = null;
+		try {
+			query = parser.parse(queryStr);
+
+		} catch (ParseException e2) {
+
+			return results;
+		}
+		TopDocs hits = null; // TopDocs search(Query query, int n)
+		// Finds the top n hits for query.
+		try {
+			hits = indexSearcher
+					.search(query, MAX_DOCS_TO_USE_FOR_CLASSIFY + 2);
+		} catch (IOException e1) {
+			logger.error("problem searching index \n" + e1);
+		}
+		logger.debug("Found " + hits.totalHits + " hits for " + queryStr);
+		int count = 0;
+		
+
+		for (ScoreDoc scoreDoc : hits.scoreDocs) {
+			Document doc = null;
+			try {
+				doc = indexSearcher.doc(scoreDoc.doc);
+			} catch (IOException e) {
+				logger.error("Problem searching training set for classif \n"
+						+ e);
+				continue;
+			}
+			String flag = doc.get("class");
+
+			Float scoreForClass = scoredClasses.get(flag);
+			if (scoreForClass == null)
+				scoredClasses.put(flag, scoreDoc.score);
+			else
+				scoredClasses.put(flag, scoreForClass + scoreDoc.score);
+
+			logger.debug(" <<categorized as>> " + flag + " | score="
+					+ scoreDoc.score + " \n text =" + doc.get("text") + "\n");
+
+			if (count > MAX_DOCS_TO_USE_FOR_CLASSIFY) {
+				break;
+			}
+			count++;
+		}
+		try {
+			scoredClasses = ValueSortMap.sortMapByValue(scoredClasses, false);
+			List<String> resultsAll = new ArrayList<String>(
+					scoredClasses.keySet()), resultsAboveThresh = new ArrayList<String>();
+			for (String key : resultsAll) {
+				if (scoredClasses.get(key) > MIN_TOTAL_SCORE_FOR_CATEGORY)
+					resultsAboveThresh.add(key);
+				else
+					logger.debug("Too low score of " + scoredClasses.get(key)
+							+ " for category = " + key);
+			}
+
+			int len = resultsAboveThresh.size();
+			if (len > MAX_CATEG_RESULTS)
+				results = resultsAboveThresh.subList(0, MAX_CATEG_RESULTS); // get
+			// maxRes
+			// elements
+			else
+				results = resultsAboveThresh;
+		} catch (Exception e) {
+			logger.error("Problem aggregating search results\n" + e);
+		}
+		if (results.size() < 2)
+			return results;
+
+		// if two categories, one is very high and another is relatively low
+		if (scoredClasses.get(results.get(0))
+				/ scoredClasses.get(results.get(1)) > BEST_TO_NEX_BEST_RATIO) // second
+			// best
+			// is
+			// much
+			// worse
+			return results.subList(0, 1);
+		else
+			return results;
+
+	}
+
+	
+
+	
+	public static String formClassifQuery(String pageContentReader, int maxRes) {
+
+		// We want to control which delimiters we substitute. For example '_' &
+		// \n we retain
+		pageContentReader = pageContentReader.replaceAll("[^A-Za-z0-9 _\\n]",
+				"");
+
+		Scanner in = new Scanner(pageContentReader);
+		in.useDelimiter("\\s+");
+		Map<String, Integer> words = new HashMap<String, Integer>();
+
+		while (in.hasNext()) {
+			String word = in.next();
+			if (!StringUtils.isAlpha(word) || word.length() < 4)
+				continue;
+
+			if (!words.containsKey(word)) {
+				words.put(word, 1);
+			} else {
+				words.put(word, words.get(word) + 1);
+			}
+		}
+		in.close();
+		words = ValueSortMap.sortMapByValue(words, false);
+		List<String> resultsAll = new ArrayList<String>(words.keySet()), results = null;
+
+		int len = resultsAll.size();
+		if (len > maxRes)
+			results = resultsAll.subList(len - maxRes, len - 1); // get maxRes
+			// elements
+			else
+				results = resultsAll;
+
+		return results.toString().replaceAll("(\\[|\\]|,)", " ").trim();
+	}
+
+	public void close() {
+		try {
+			indexReader.close();
+		} catch (IOException e) {
+			logger.error("Problem closing index \n" + e);
+		}
+	}	
+	
+	
+	/*
+	 * Main entry point for classifying sentences
+	 */
+
+	public List<String> getEntityOrClassFromText(String content) {
+
+		List<String> sentences = TextProcessor.splitToSentences(content);
+		List<String> classifResults;
+
+		try {
+			for (String sentence : sentences) {
+				// If sentence is too short, there is a chance it is not form a
+				// main text area,
+				// but from somewhere else, so it is safer not to use this
+				// portion of text for classification
+
+				if (sentence.length() < MIN_SENTENCE_LENGTH_TO_CATEGORIZE)
+					continue;
+				String query = formClassifQuery(sentence, MAX_TOKENS_TO_FORM);
+				classifResults = classifySentence(query);
+				if (classifResults != null && classifResults.size() > 0) {
+					for (String c : classifResults) {
+						localCats.add(c);
+					}
+					logger.debug(sentence + " =>  " + classifResults);
+				}
+			}
+
+		} catch (Exception e) {
+			logger.error("Problem classifying sentence\n " + e);
+		}
+		
+		List<String> aggrResults = new ArrayList<String>();
+		try {
+
+			aggrResults = localCats.getFrequentTags();
+
+			logger.debug(localCats.getFrequentTags());
+		} catch (Exception e) {
+			logger.error("Problem aggregating search results\n" + e);
+		}
+		return aggrResults;
+	}
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/doc_classifier/DocClassifierTrainingSetMultilingualExtender.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/doc_classifier/DocClassifierTrainingSetMultilingualExtender.java b/opennlp-similarity/src/main/java/opennlp/tools/doc_classifier/DocClassifierTrainingSetMultilingualExtender.java
new file mode 100644
index 0000000..73b0d43
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/doc_classifier/DocClassifierTrainingSetMultilingualExtender.java
@@ -0,0 +1,284 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.doc_classifier;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.channels.Channels;
+import java.nio.channels.ReadableByteChannel;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+
+import opennlp.tools.similarity.apps.BingQueryRunner;
+import opennlp.tools.similarity.apps.utils.PageFetcher;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.tika.Tika;
+import org.apache.tika.exception.TikaException;
+import org.json.JSONObject;
+
+/*
+ * This utility gets 'training_corpus' as input and creates a new version of training_corpus with verified files.
+ * Verified => classified by existing training set as only belonging to its target category, no other categories, not empty.
+ */
+public class DocClassifierTrainingSetMultilingualExtender {
+	private static final String LANG_TEMPL = "l_a_n_g";
+	private String wikiUrlsTemplate = "https://"+LANG_TEMPL+".wikipedia.org/wiki/";
+	
+	public static String projectHome = new File(".").getAbsolutePath().replace("contentinspection/.", "");
+	public static String resourceDir = new File(".").getAbsolutePath().replace("/.", "") + "/src/main/resources";
+	DocClassifier classifier = null;
+	private String sourceDir = null, destinationDir = null;
+	//interwiki-fr"><a href="http://fr.wikipedia.org/wiki/Niveau_d%27%C3%A9nergie" title="Niveau d&#39;�nergie \u2013 French" lang="fr" 
+	private static String[][] multilingualTokens = new String[][]{ 
+		{"interwiki-fr\"><a href=\"", "lang=\"fr\""},
+		{"interwiki-es\"><a href=\"", "lang=\"es\""},
+		{"interwiki-de\"><a href=\"", "lang=\"de\""},
+	};
+	
+	private static String[] langs = new String[]{ "fr", "es", "de"};
+
+	protected ArrayList<File> queue = new ArrayList<File>();
+
+	protected Tika tika = new Tika();
+	public DocClassifierTrainingSetMultilingualExtender(String resource) {
+
+		classifier = new DocClassifier("", new JSONObject());
+
+	}
+	private int FRAGMENT_LENGTH = 500;
+	
+
+	protected void addFiles(File file) {
+
+		try {
+			if (!file.exists()) {
+				System.out.println(file + " does not exist.");
+			}
+			if (file.isDirectory()) {
+				for (File f : file.listFiles()) {
+					try {
+						addFiles(f);
+					} catch (Exception e) {
+					}
+				}
+			} else {
+				queue.add(file);
+			}
+		} catch (Exception e) {
+
+		}
+	}
+	
+	public List<String> extractEntriesFromSpecial_Export(String filename){
+		List<String> filteredEntries = new ArrayList<String>();
+		String content=null;
+		try {
+			content = FileUtils.readFileToString(new File(filename));
+		} catch (IOException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		}
+		String[] entries = StringUtils.substringsBetween(content, "[[", "]]");
+		for(String e: entries){
+			if (e.startsWith("Kategorie") || e.startsWith("Category") || e.startsWith("d:") || e.startsWith("User") 
+					||e.startsWith("Portal")  )
+				continue;
+			if (e.indexOf(':')>-1)
+				continue;
+			
+			if (e.indexOf(":")>-1)
+				continue;
+			int endofEntry = e.indexOf('|');
+			if (endofEntry>-1) e = e.substring(0, endofEntry);
+			filteredEntries.add(e);
+		}
+		
+		filteredEntries = new ArrayList<String> (new HashSet<String>(filteredEntries));
+		return filteredEntries;
+	}
+
+	public void processDirectory(String fileName) throws IOException {
+		List<String[]> report = new ArrayList<String[]>();
+		report.add(new String[] { "filename", "category",
+				"confirmed?" ,
+		});
+		
+		addFiles(new File(fileName));
+	//	FileUtils.deleteDirectory(new File(destinationDir));
+	//	FileUtils.forceMkdir(new File(destinationDir));
+		
+
+		for (File f : queue) {
+			String content = null;
+			try {// should be wiki page
+				//if (f.getName().toString().toLowerCase().indexOf(" wiki")<0 && 
+						
+			//	if (		f.getAbsolutePath().indexOf("wiki-new")<0)
+			//		continue;
+				// should not be a page already derived by a link
+				if (f.getName().toString().toLowerCase().indexOf(".html_")>-1)
+					continue;
+				
+				System.out.println("processing "+f.getName());
+				content = FileUtils.readFileToString(f, "utf-8");
+				int langIndex =0;
+				for(String[] begEnd: multilingualTokens){
+					String urlDirty = StringUtils.substringBetween(content, begEnd[0], begEnd[1]);
+					String url = StringUtils.substringBefore(urlDirty, "\"");
+
+					if (url!=null){
+						if (!url.startsWith("http:"))
+						    url = "https:"+url;
+						
+						String[] parts  = url.split("/");
+						String multilingualName = parts[parts.length-1];
+						String destFileName = f.getAbsolutePath().replace(sourceDir, destinationDir).replace(" - Wikipedia, the free encyclopedia.html", "-wiki")+"."+langs[langIndex]+"."
+								+"_"+multilingualName+".html";
+						if (!new File(destFileName).exists()){
+							saveDocFromTheWeb(url, destFileName);
+							System.out.println(f.getName()+ " => "+destFileName);
+						}
+					} else {
+						System.out.println("Unable to extract multilingual urls for'" +langs[langIndex] +"' from file "+ f.getCanonicalPath());
+					}
+					langIndex++;
+				}
+			} catch (Exception ee) {
+				ee.printStackTrace();
+			}
+		}
+
+
+		queue.clear();
+	}
+
+	private void copyURLToFile(URL url, File file) {
+		ReadableByteChannel rbc=null;
+		try {
+			rbc = Channels.newChannel(url.openStream());
+		} catch (IOException e1) {
+			// TODO Auto-generated catch block
+			e1.printStackTrace();
+		}
+		FileOutputStream fos=null;
+		try {
+			fos = new FileOutputStream(file.getAbsolutePath());
+		} catch (FileNotFoundException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		}
+		try {
+			fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE);
+		} catch (IOException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		}
+		
+	}
+	
+	public void crawlWikiOnTopic( String filename, String lang, String destinationDir){
+		List<String> entries = extractEntriesFromSpecial_Export(filename);
+		for(String e: entries){
+			String url  = wikiUrlsTemplate.replace(LANG_TEMPL, lang) + e; 
+			saveDocFromTheWeb(url, destinationDir+e.replace(' ', '_')+".html"); 
+		}
+	}
+	
+	public static void saveDocFromTheWeb(String docUrl, String destinationFile) {
+		try {
+			URL url = new URL(docUrl);
+			InputStream is = url.openStream();
+			if (!new File(destinationFile).exists()) {
+				new File(destinationFile).createNewFile();
+			}
+
+			OutputStream os = new FileOutputStream(destinationFile);
+
+
+			byte[] b = new byte[2048];
+			int length;
+
+			while ((length = is.read(b)) != -1) {
+				os.write(b, 0, length);
+			}
+
+			is.close();
+			os.close();
+		} catch (MalformedURLException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		} catch (FileNotFoundException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		} catch (IOException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		}
+	}
+	
+
+	public static void main(String[] args) {
+		if (args.length < 2) {
+			System.err
+			.println("Verifier accepts two arguments: [0] - input 'training_corpus' folder, "
+					+ "[1] - output 'training_corpus' folder . "
+					+ "All paths should include category name as a part of full path string, such as '/computing/' " );
+			System.exit(0);
+		}
+
+		DocClassifierTrainingSetMultilingualExtender runner = new DocClassifierTrainingSetMultilingualExtender(null);
+		
+		if (args.length==2) {
+		runner.sourceDir = args[0]; runner.destinationDir = args[1];
+		runner.sourceDir =
+				"/Users/borisgalitsky/Documents/svm_tk_july2015/milkyway/training_corpus_multilingual_verif";
+		runner.destinationDir =
+				"/Users/borisgalitsky/Documents/new_corpus/milkyway/training_corpus_new_multilingual_refined";
+
+		try {
+			runner.processDirectory( runner.sourceDir);
+		} catch (IOException e) {
+			e.printStackTrace();
+		}
+		} else {  
+			runner.crawlWikiOnTopic("/Users/borisgalitsky/Downloads/Wikipedia-20150730124756.xml",
+					//Wikipedia-20150730053619.xml",
+					////Wikipedia-20150730044602.xml",
+					//Wikipedia-20150729103933.xml",
+					//Wikipedia-20150729103933.xml",
+					// "Wikipedia-20150728193126.xml",
+					//Wikipedia-20150728183128.xml",
+					"en", 
+					"/Users/borisgalitsky/Documents/merged_svm_tk/milkyway/training_corpus_new_multilingual/business/wiki/wiki-new/");
+		}
+
+
+	}
+}
+
+/*
+/Users/borisgalitsky/Documents/workspace/deepContentInspection/src/main/resources/docs/netflix
+ */
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/doc_classifier/DocClassifierTrainingSetVerifier.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/doc_classifier/DocClassifierTrainingSetVerifier.java b/opennlp-similarity/src/main/java/opennlp/tools/doc_classifier/DocClassifierTrainingSetVerifier.java
new file mode 100644
index 0000000..98debe6
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/doc_classifier/DocClassifierTrainingSetVerifier.java
@@ -0,0 +1,163 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.doc_classifier;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import opennlp.tools.jsmlearning.ProfileReaderWriter;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.tika.Tika;
+import org.apache.tika.exception.TikaException;
+import org.json.JSONObject;
+
+/*
+ * This utility gets 'training_corpus' as input and creates a new version of training_corpus with verified files.
+ * Verified => classified by existing training set as only belonging to its target category, no other categories, not empty.
+ */
+public class DocClassifierTrainingSetVerifier {
+	public static String projectHome = new File(".").getAbsolutePath();
+	public static String resourceDir = new File(".").getAbsolutePath().replace("/.", "") + "/src/main/resources";
+	DocClassifier classifier = null;
+	private String sourceDir = null, destinationDir = null;
+	
+
+	protected ArrayList<File> queue = new ArrayList<File>();
+
+	protected Tika tika = new Tika();
+	public DocClassifierTrainingSetVerifier(String resource) {
+
+		
+		classifier = new DocClassifier("", new JSONObject());
+
+	}
+	private int FRAGMENT_LENGTH = 500;
+
+
+	protected void addFiles(File file) {
+
+		try {
+			if (!file.exists()) {
+				System.out.println(file + " does not exist.");
+			}
+			if (file.isDirectory()) {
+				for (File f : file.listFiles()) {
+					try {
+						addFiles(f);
+					} catch (Exception e) {
+					}
+				}
+			} else {
+				queue.add(file);
+			}
+		} catch (Exception e) {
+
+		}
+	}
+
+	public void processDirectory(String fileName) throws IOException {
+		List<String[]> report = new ArrayList<String[]>();
+		report.add(new String[] { "filename", "category",
+				"confirmed?" ,
+		});
+		
+		addFiles(new File(fileName));
+		//FileUtils.deleteDirectory(new File(destinationDir));
+		//FileUtils.forceMkdir(new File(destinationDir));
+		
+
+		for (File f : queue) {
+			String content = null;
+			try {
+				System.out.println("processing "+f.getName());
+				
+				//if (f.getName().indexOf(".html")<0)
+					//continue;
+				classifier = new DocClassifier("", new JSONObject());
+
+
+				content = tika.parseToString(f);
+
+				//classifier.runExpressionsOnContent(content);
+				List<String> resultsClassif = classifier.getEntityOrClassFromText(content);
+				Boolean bRejected = true;
+				if (resultsClassif.size()==1 
+						&& resultsClassif.get(0).equals(
+								ClassifierTrainingSetIndexer.getCategoryFromFilePath(f.getAbsolutePath()))){
+					String destFileName = f.getAbsolutePath().replace(sourceDir, destinationDir);
+					FileUtils.copyFile(f, new File(destFileName));
+					bRejected = false;
+				} else {
+					System.out.println("File "+ f.getAbsolutePath() + "\n classified as "+
+							resultsClassif.toString() + " but should be " + ClassifierTrainingSetIndexer.getCategoryFromFilePath(f.getAbsolutePath()) );
+				}
+				bRejected = !bRejected;
+				String fragment = content;
+				if (content.length() > FRAGMENT_LENGTH)
+					fragment = content.substring(0, FRAGMENT_LENGTH);
+				fragment = fragment.replaceAll("\n", " ").trim();
+				report.add(new String[] { f.getName(),  resultsClassif.toString(), ClassifierTrainingSetIndexer.getCategoryFromFilePath(f.getAbsolutePath()),
+						(bRejected).toString(),   
+						fragment});
+				ProfileReaderWriter.writeReport(report,  "DocClassifierMultiLingRpt.csv");
+
+			} catch (TikaException e) {
+				System.out.println("Tika problem with file"
+						+ f.getAbsolutePath());
+			} catch (Exception ee) {
+				ee.printStackTrace();
+			}
+		}
+
+
+		queue.clear();
+	}
+
+	public static void main(String[] args) {
+		if (args.length < 2) {
+			System.err
+			.println("Verifier accepts two arguments: [0] - input 'training_corpus' folder, "
+					+ "[1] - output 'training_corpus' folder . "
+					+ "All paths should include category name as a part of full path string, such as '/computing/' " );
+			System.exit(0);
+		}
+
+		DocClassifierTrainingSetVerifier runner = new DocClassifierTrainingSetVerifier(null);
+		runner.sourceDir = args[0]; runner.destinationDir = args[1];
+		runner.sourceDir =
+			//	"/Users/borisgalitsky/Documents/svm_tk_july2015/milkyway/eval_corpus_multiling";
+				"/Users/borisgalitsky/Documents/merged_svm_tk/milkyway/training_corpus_new_multilingual";
+		runner.destinationDir =
+				"/Users/borisgalitsky/Documents/svm_tk_july2015/milkyway/training_corpus_multilingual_verif";
+		//	"/Users/borisgalitsky/Documents/svm_tk_july2015/milkyway/eval_corpus_multiling_bogus";
+
+		try {
+			runner.processDirectory( runner.sourceDir);
+		} catch (IOException e) {
+			e.printStackTrace();
+		}
+
+
+	}
+}
+
+/*
+/Users/borisgalitsky/Documents/workspace/deepContentInspection/src/main/resources/docs/netflix
+ */
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/enron_email_recognizer/EmailNormalizer.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/enron_email_recognizer/EmailNormalizer.java b/opennlp-similarity/src/main/java/opennlp/tools/enron_email_recognizer/EmailNormalizer.java
new file mode 100644
index 0000000..01aaa12
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/enron_email_recognizer/EmailNormalizer.java
@@ -0,0 +1,100 @@
+package opennlp.tools.enron_email_recognizer;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+
+import org.apache.commons.io.FileUtils;
+
+public class EmailNormalizer {
+	protected ArrayList<File> queue = new ArrayList<File>();
+	
+	protected void addFilesPos(File file) {
+
+		if (!file.exists()) {
+			System.out.println(file + " does not exist.");
+		}
+		if (file.isDirectory()) {
+			for (File f : file.listFiles()) {
+				addFilesPos(f);
+				System.out.println(f.getName());
+			}
+		} else {
+			queue.add(file);
+		}
+	}
+	
+	public static final String[] headers = new String[] {
+		"Message-ID:",
+	"Date:",
+	"From:",
+	"To:",
+	"Subject:",
+	"Mime-Version:",
+	"Content-T",
+	"X-From:",
+	"X-To:",
+	"X-cc:",
+	"X-bcc:",
+	"X-Folder:",
+	"X-Origin:",
+	"X-FileName",
+	"cc:",
+	"----",
+	};
+	
+	public static final String[] prohibitedStrings = new String[] {
+		"@", "<", ">"
+	};
+
+	private String OrigFolder = "maildir_ENRON_EMAILS", NewFolder = "data";
+
+	
+	
+	public void normalizeAndWriteIntoANewFile(File f){
+		String content="";
+        try {
+	        content = FileUtils.readFileToString(f);
+        } catch (IOException e) {
+	        // TODO Auto-generated catch block
+	        e.printStackTrace();
+        }
+		String[] lines = content.split("\n");
+		StringBuffer buf = new StringBuffer();
+		for(String l: lines){
+			boolean bAccept = true;
+			for(String h: headers){
+				if (l.startsWith(h)){
+					bAccept = false;
+				}
+			}
+			for(String h: prohibitedStrings){
+				if (l.indexOf(h)>0){
+					bAccept = false;
+				}
+			}
+			if (bAccept)
+				buf.append(l+"\n");
+		}
+		String directoryNew = f.getAbsolutePath().replace(OrigFolder, NewFolder);
+		try {
+			String fullFileNameNew = directoryNew +"txt";
+	        FileUtils.writeStringToFile(new File(fullFileNameNew), buf.toString());
+        } catch (IOException e) {
+	        // TODO Auto-generated catch block
+	        e.printStackTrace();
+        }
+	}
+	
+	public void normalizeDirectory(File f){
+		addFilesPos(f);
+		for(File e: queue){
+			normalizeAndWriteIntoANewFile(e);
+		}
+	}
+	
+	public static void main(String[] args){
+		EmailNormalizer nrm = new EmailNormalizer();
+		nrm.normalizeDirectory(new File("/Users/bgalitsky/Documents/ENRON/maildir_ENRON_EMAILS"));
+	}
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/enron_email_recognizer/EmailTrainingSetFormer.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/enron_email_recognizer/EmailTrainingSetFormer.java b/opennlp-similarity/src/main/java/opennlp/tools/enron_email_recognizer/EmailTrainingSetFormer.java
new file mode 100644
index 0000000..9cb713f
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/enron_email_recognizer/EmailTrainingSetFormer.java
@@ -0,0 +1,67 @@
+package opennlp.tools.enron_email_recognizer;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.List;
+
+import org.apache.commons.io.FileUtils;
+
+public class EmailTrainingSetFormer {
+	static String dataDir = "/Users/bgalitsky/Downloads/",
+			//enron_with_categories/",
+			fileListFile = "cats4_11-17.txt",
+			destinationDir = "/Users/bgalitsky/Documents/ENRON/data11_17/";
+
+	//enron_with_categories/5/70665.cats:4,10,1
+	public static void  createPosTrainingSet(){
+		try {
+			List<String> lines = FileUtils.readLines(new File(dataDir+fileListFile));
+			for(String l: lines){
+				Integer endOfFname = l.indexOf('.'),
+						startOfFname = l.lastIndexOf('/');
+				String filenameOld =dataDir+ l.substring(0, endOfFname)+".txt";
+
+				String content = normalize(new File(filenameOld));
+
+				String filenameNew = destinationDir  + l.substring(startOfFname+1, endOfFname)+".txt";
+				//FileUtils.copyFile(new File(filenameOld), new File(filenameNew));
+				FileUtils.writeStringToFile(new File(filenameNew), content);
+			}
+		} catch (Exception e) {
+			e.printStackTrace();
+		}
+	}
+
+	private String origFolder = "maildir_ENRON_EMAILS", newFolder = "data11_17";
+
+	public static String normalize(File f){
+		String content="";
+		try {
+			content = FileUtils.readFileToString(f);
+		} catch (IOException e) {
+			e.printStackTrace();
+		}
+		String[] lines = content.split("\n");
+		StringBuffer buf = new StringBuffer();
+		for(String l: lines){
+			boolean bAccept = true;
+			for(String h: EmailNormalizer.headers){
+				if (l.startsWith(h)){
+					bAccept = false;
+				}
+			}
+			for(String h: EmailNormalizer.prohibitedStrings){
+				if (l.indexOf(h)>0){
+					bAccept = false;
+				}
+			}
+			if (bAccept)
+				buf.append(l+"\n");
+		}
+		return buf.toString();
+	}
+
+	public static void main(String[] args){
+		EmailTrainingSetFormer.createPosTrainingSet();
+	}
+}


[21/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/109TeleHTC_Manual_2_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/109TeleHTC_Manual_2_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/109TeleHTC_Manual_2_EN.txt.txt
new file mode 100644
index 0000000..031b60f
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/109TeleHTC_Manual_2_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Chapter 2 Using Phone Features 2.1 Making Calls To make a call , you can either dial a number or select a contact you want to call . Note Most SIM cards are preset with a PIN ( personal identification number ) that is provided by your wireless service provider . When prompted , enter the preset PIN , then tap Enter . If you enter the wrong PIN three times , the SIM card is blocked . If this happens , you can unblock it with the PUK ( PIN Unblocking Key ) obtained from your wireless service provider . Making a call from the Phone screen To open the Phone screen , press the TALK/SEND button ( ) . The Phone\u2019s Smart Dial feature makes it quick and easy for you to place a call . When you start entering a phone number or contact name , Smart Dial automatically searches and sorts the contacts on the SIM and in Contacts , and the phone numbers in Call History ( incoming , outgoing , and missed calls ) . You can then select the desired number or contact from the filtered list . You can jus
 t enter the first letter of the name or number of a contact . Smart Dial searches for contact phone numbers and the first character of a contact name as well as from the character that appears after a space , dash , or underscore in a contact name . 46 Using Phone Features To call a phone number or contact 1. Begin entering the first few numbers or characters by tapping the keys on the Phone keypad . As you enter numbers or letters , the Smart Dial panel displays matches found . If you tap a wrong number or character , tap to erase each subsequent digit or character . To erase the entire entry , tap and hold . 2. Tap to close the phone keypad and to see if there are more matching numbers or contacts in the Smart Dial panel . To browse through the filtered list , flick through the list or slide your finger up or down the list slowly . Add the number to your contacts . Smart Dial panel Open the contact details . Tap to open or close the phone keypad . 3. To call a number or contact , 
 do one of the following : \u2022 Tap the desired phone number or contact from the list . \u2022 To call a different phone number associated with the contact , touch the contact card icon at the right side of the contact name . On the contact details screen , tap the number you want to call . Using Phone Features 47 Tips \u2022 During a call , use the VOLUME UP/DOWN button on the side of the device to adjust the phone volume . \u2022 To send a text message to a selected contact , touch , and then tap Send Message . For more information about sending text messages , see \u201c Text Messages \u201d in Chapter 6. Making a call from a text message While you are viewing a text message on the TouchFLO 3D Messages tab or in the Inbox , you can directly call the sender of the message . 1. While viewing the text message , press the TALK/SEND button . 2. The Phone screen then opens , with the sender\u2019s phone number automatically filled in and ready to be dialed . To place the call , tap Talk on the Phone scree
 n or press TALK/SEND again . Tapping a phone number that is displayed in the body of the text message also allows you to call that number . For more information about viewing text messages on the TouchFLO 3D Messages tab , see \u201c Messages \u201d in Chapter 3. For more information about working with text messages , see \u201c Text Messages \u201d in Chapter 6. Making a call from an e-mail message While viewing a received e-mail message , you can directly place a call to the sender of the message . Tapping a phone number that is displayed in the body of the e-mail message also allows you to call that number . Note To receive e-mail , set up your e-mail account first on your device . See \u201c Mail \u201d in Chapter 3 for instructions . From the TouchFLO 3D Mail tab or Inbox When you are browsing e-mail messages on the TouchFLO 3D Mail tab or in the Inbox , you can select a received e-mail and then directly call the sender by pressing the TALK/SEND button . To call the sender , follow the same step
 s in \u201c Making a call from a text message \u201d . 48 Using Phone Features For more information about browsing through e-mail on the TouchFLO 3D Mail tab , see \u201c Mail \u201d in Chapter 3. For more information about working with e-mail messages , see Chapters 6 and 7. From an open e-mail When you open a received e-mail , you can directly place a call to the sender . 1. On the TouchFLO 3D Home screen , slide to the Mail tab . 2. Swipe your finger up or down the screen to browse through received messages . Tap an e-mail message to open it . 3. To place a call , you can do the following : \u2022 If the sender is a contact that is stored with a phone number on your device , tap Talk ( ) that is displayed to the right of the sender name to directly dial the sender\u2019s phone number . \u2022 To call the phone number that is contained in the e-mail message , tap that phone number . \u2022 Tap the To or Cc button to open the Select Contacts to Call screen , and then tap Phone to open the Phone screen . T
 alk button Using Phone Features 49 Tip You can also start a conference call from the Select Contacts to Call screen . See \u201c Starting a conference call from an e-mail \u201c later in this chapter for details . 4. If you made a call to the sender or the phone number contained in the e-mail message , you can directly view the e-mail message on the Phone screen while the call is in progress . When the call is made from an e-mail message , an e-mail indicator is shown on the Keypad button . Tap this button to view the message directly on the Phone screen . Making a call from a meeting request or reminder When you receive a calendar reminder or meeting request e-mail that contains a phone number , you can tap the number to directly place the call . For example , if you are traveling and need to join a conference bridge call back in the home office , rest assured that you can join the call without delay . Appointment reminders you receive on your device will conveniently display the phone n
 umber and PIN , ready for you to dial . 50 Using Phone Features If you made a call from a meeting request e-mail , you can view the meeting details in the e-mail directly on the Phone screen while the call is in progress . An e-mail indicator is shown on the Keypad button ( ) . Tap this button to view the meeting details . Making an emergency call Tip Enter the international emergency number for your locale , and press the TALK/SEND button on your device . Additional emergency numbers may be included in your SIM card . Contact your wireless service provider for details . Making an international call 1. Tap and hold the number 0 key on the Phone screen until the plus ( + ) sign appears . The plus ( + ) sign replaces the international prefix of the country that you are calling . 2. Enter the full phone number you want to dial , and press the TALK/SEND button on your device . The full phone number includes the country code , area code ( without the leading zero , if any ) , and phone n
 umber . Calling voice mail Number 1 is generally reserved for your voice mail . Tap and hold this button on the Phone screen keypad to call voice mail and retrieve your voice mail messages . Using Phone Features 51 2.2 Answering and Ending Calls When you receive a phone call , the Incoming Voice Call screen will appear , allowing you to either answer or ignore the call . Accepting or rejecting an incoming voice call To answer a call Do any of the following : \u2022 \u2022 Fully slide the center button to the left of the Answer/Ignore bar . Press the TALK/SEND button . To reject a call Do any of the following : \u2022 \u2022 Fully slide the center button to the right of the Answer/Ignore bar . Press the END button . Center button To mute the ringing sound To mute the ring without rejecting the call , tap Mute Ring or press the VOLUME DOWN button . You can also put the device face down on a flat surface to mute the ringing sound . To reject a call and send a text message You can automatically sen
 d a default text message to the caller and reject the incoming voice call . Tap Send Text when you have an incoming call to send the text message . You can change the default text message by tapping Menu > Options > Advanced tab while on the Phone screen . 52 Using Phone Features Using the call functions When a call is in progress , use the call function buttons to put a call on hold , dial another contact for a phone conference , enter notes about the conversation , and more . Call functions Contact details . See \u201c The Contact Details screen \u201d in Chapter 3. To swap between two calls Your device notifies you when you have another incoming call , and gives you the option of rejecting or accepting the call . If you are already on a call and accept the new call , you can choose to switch between the two callers . 1. Press the TALK/SEND button to accept the second call and put the first call on hold . 2. To switch between callers , tap or tap the dim area that shows \u201c On Hold \u201d 
 . 3. To end the second call and return to the first call , tap End Call or press END on your device . Using Phone Features 53 To use the speakerphone During a call , do one of the following to turn the speakerphone on or off : \u2022 \u2022 \u2022 Note Tap to turn the speakerphone on . Tap this button again to turn the speakerphone off . Placing your device face down automatically turns on the speakerphone . To turn the speakerphone off , place your device face up . You can also press and hold the TALK/SEND button during a call to turn the speakerphone on or off . The speakerphone icon ( ) appears in the title bar when the speakerphone is on . Warning ! avoid damage to your hearing , do not hold the device against your ear when To the speakerphone is on . To mute the microphone during a call Tap to turn the microphone on or off . When the microphone is turned off , the mute icon appears on the screen . To turn noise reduction on or off When you are using the earpiece and not the speakerphone
  during a call , you can tap Menu > Noise Reduction On to reduce ambient noise if the other party cannot hear your voice clearly . When noise reduction is not needed , tap Menu > Noise Reduction Off to disable it and save battery power . You can also turn noise reduction on or off in the Phone settings . Tap Start > Settings > All Settings > Personal tab > Phone > Advanced tab and then select one of the following settings in the Reduce noise in phone calls drop-down list : \u2022 Automatically . Analyzes ambient noise only at the beginning of a connected call and applies a certain noise reduction setting based on the detected noise . However , when you move to a noisier environment during the call , your device will not further decrease noise based on your new location . 54 Using Phone Features \u2022 \u2022 Notes Always . Always reduces ambient noise during the entire course of your phone call . Never . Turns off noise reduction . \u2022 If you want to change the noise reduction setting to Aut
 omatically , make sure to do this before starting a call . \u2022 Noise reduction is not used when you have a wired or Bluetooth headset connected . To open Notes while in a call Tap , or pull out the stylus . To view or create appointments during a call Tap to open the Calendar and browse your appointments or create a new one . Close the Calendar screen to return to the Phone screen . To end a call When a call is in progress , tap End Call or press END . Using Phone Features 55 2.3 Using Call History The missed call icon icon appears in the title bar when you missed a call . Use the Call History tab to check who the caller was , or view all your dialed numbers , and received calls . To open the Call History tab , tap Call History or Missed Call on the TouchFLO 3D Home tab . Received call Dialed number Missed call Return to the People tab . Open the Call History tab of the Contact Details screen and view all calls you exchanged with the contact . Filter or clear the Call History tab . 
 If you have a long list of calls in the Call History tab , tap Menu > Filter , and then select the type of call you want to display . 56 Using Phone Features 2.4 Speed Dial Use Speed Dial to call frequently-used numbers with a single tap . For example , if you assign a contact to the location 2 in Speed Dial , you can tap and hold the number 2 key on the Phone screen keypad to dial the contact\u2019s number . To create a Speed Dial entry 1. On the Phone screen , tap Menu > Speed Dial . 2. Tap Menu > New . 3. Tap a contact , and then select the phone number of the contact that you want to the speed dial key . 4. In Location , select an available key to use as the speed dial key , then tap OK . Tip To delete a Speed Dial entry , in the Speed Dial list , tap and hold the desired entry , and then tap Delete . 2.5 Video Call Your device lets you make video calls when you are within a 3G network coverage area . You need to use a 3G-enabled SIM card and activate the video calling service on y
 our call plan to make video calls . Making a video call 1. On the Phone screen , tap to switch to video call mode . 2. Begin entering the first few digits of the phone number or the first few letters of the contact name you want to call . You can also tap to close the phone keypad so you can simply finger-scroll and select from the list of contacts and phone numbers . 3. After entering or selecting a phone number or contact name , tap the Video Call button at the bottom of the screen . Using Phone Features 57 4. When connected , tap the Camera button , and then wait for the called party\u2019s image to appear on the Phone screen . Notes \u2022 The signal strength may influence the quality of streaming images . \u2022 The speakerphone automatically turns on when a video call is established . \u2022 If the video call is unsuccessful , a message pops up asking you if you want to try again , make a voice call instead , or send an SMS/MMS message . Changing the video call settings 1 While on a video
  call , you can change certain settings such as the size and brightness of the image , and the camera to use . Displays your image and the other party 's image , depending on the selected Swap View setting . Turns the camera on or off . When the camera is turned off , the image displayed depends on the Video Call options you have set . See \u201c To change the video call options \u201d for details . Switches between the Main ( back ) and Second ( front ) camera . Switches between 2 view options . You can choose to have the other party\u2019s image small or large . Displays the phone number or contact name and call timer . 2 1 3 5 2 3 4 4 5 58 Using Phone Features To adjust the brightness 1. During a video call , tap Menu > Brightness . A brightness bar appears on the screen . 2. Drag the slider to the right to increase the brightness or to the left to decrease the brightness of the live image . Note You cannot adjust the brightness for a still image . To change the video call options On the 
 Phone screen , or on the screen of the ongoing video call , tap Menu > Video Call Options and set the following : General tab \u2022 Show still image . Select the image you want to display to the other party when you turn off the camera . \u2022 Preset ( female ) . Displays a female picture . \u2022 Preset ( male ) . Displays a male picture . \u2022 Custom image . Use an image file as your display picture . When this option is selected , tap the Browse button ( ) to select an image file . \u2022 Last video frame . Use the last video frame before you stopped the camera as your display image . \u2022 Video screen preference . Choose the default image sizes to display on your device . \u2022 Camera options : \u2022 Flip the 2nd Camera video . Select this option if you are going to show a text document to the other party using the second camera . \u2022 Switch camera off when answering video call / Switch camera off when making video call . Display a still image instead of live video to the other party when you a
 nswer or make a video call . The image displayed is the one selected in Show still image . Using Phone Features 59 \u2022 Display info : \u2022 Show call timer . Display the elapsed call time during a call . \u2022 Show phone number or name . Display the other party\u2019s phone number or name during a call . Services tab To set up further calling options for Video Call , tap the Services tab . Accepting or rejecting an incoming video call When you receive a video call , the Incoming Video Call screen appears . Refer to the \u201c Accepting or rejecting an incoming voice call \u201d section earlier in this chapter for details on how to accept or reject an incoming video call , mute the ringing sound , and more . 2.6 Straight Talk\u2122 Your device features Straight Talk\u2122 , a technology that integrates messaging , voice , and conference calling , allowing you to seamlessly transition from a message to a single call or multi-party conference call . With dual microphone and speaker that enhance voice qua
 lity and reduce noise , Straight Talk turns your device into a conference room speakerphone system . Note Make sure your SIM card is enabled with conference calling service . Contact your wireless service provider for details . This section shows how to make a conference call and use Straight Talk to transition from e-mail to the conference call . For more information about using Straight Talk to switch from a message to a voice call , see \u201c Making a call from a text message \u201d , \u201c Making a call from an e-mail message \u201d , and \u201c Making a call from a meeting request or reminder \u201d earlier in this chapter . 60 Using Phone Features Starting a conference call from the Phone screen There are two ways to start a conference call from the Phone screen . You can choose your participants first before starting the conference call . You can also directly dial calls and add each connected call to the conference call . To select participants and start a conference call 1. Press the TALK/
 SEND button to open the Phone screen . 2. Tap to switch the Phone screen to conference mode . 3. Tap to close the phone keypad so you can finger-scroll and select from contacts or phone numbers . 4. Select conference call participants by selecting the check boxes of contacts or phone numbers , and then tap Conference . The participants list is then displayed . Tap these buttons if you want to choose from other stored phone numbers and use them to call the participants . Tip To remove a participant before starting the conference call , tap Keypad and clear the check box of this participant . Using Phone Features 61 5. Slide open the hardware keyboard so that your device will display the Conference Room screen . This is an intuitive interface that makes conference calls much easier and simpler . You can simply tap photos or buttons on the screen to dial participants , join them to your conference call , and more . 6. Tap the photo of the first participant to dial the first call . 7. W
 hen the first call is connected , tap the photo of the second participant to dial the second call . The first call is automatically put on hold . 8. When the second call is connected , tap the Join button ( conference call and then dial the next participant . ) to add it to the Repeat this same process until all participants are added and connected to the conference call . Tip If you prefer to do the conference call in portrait view , do not slide open the hardware keyboard . When in portrait view , tap the Talk button ( ) to dial each participant , and then tap the Join button ( ) to add the second call and each subsequent call to the conference . To directly dial and add each participant to the conference call 1. Press the TALK/SEND button to open the Phone screen . 2. Make the call to the first participant . See \u201c Making a call from the Phone screen \u201d for instructions . 3. When the first call is connected , tap and make the call to the second participant . The first call is p
 ut on hold when you dial the second call . 4. When the second call is connected , tap to start the conference call . 62 Using Phone Features 5. Slide open the hardware keyboard so that your device will display the Conference Room screen . 6. To add another participant , tap Add Call and make the next call . When this call is connected , tap the Join button ( ) to add it to the conference call . Repeat this same process for each subsequent call that you add to the conference call . Tip If you prefer to do the conference call in portrait view , do not slide open the hardware keyboard . When in portrait view , tap Invite participants to dial a new call , and then tap the Join button ( ) to add the connected call to the conference . Managing the conference call While a conference call is in progress , you can do the following : To manage the conference call in landscape view To talk in private with a participant and put the others in the conference call on hold , tap the participant\u2019s
  photo and then tap Private . To swap between a private call and the conference call , tap the dim area or tap Swap . Tap Join to add a private call back to the conference call . Tap End Call to end a private call and return to the conference call . Using Phone Features 63 To manage the conference call in portrait view Tap a participant from the conference call and then tap one of these buttons : View contact details . Talk in private to this participant only and put the others in the conference call on hold . End the call with this participant . To swap between a private call and the conference call , tap the dim area that shows \u201c On Hold \u201d or tap Menu > Swap . 64 Using Phone Features To turn on the speakerphone during a conference call While a call or conference call is in progress , you can turn on the speakerphone by simply placing your device face down . Your device provides crystal-clear sound quality and voice reception during conference calls . When the speakerphone is o
 n , your device also enables the dual microphone to enhance your voice quality and reduce noise . When you need to temporarily mute the dual microphone so that the conference call participants will not hear you , press the MUTE button . Press it again to unmute . To end the conference call Tap End Call or press END . Dual microphone MUTE Dual speaker Starting a conference call from an e-mail Note From a received e-mail message , you can directly start a conference call with the sender and other recipients of the e-mail . To receive e-mail , set up your e-mail account first on your device . See \u201c Mail \u201d in Chapter 3 for instructions . 1. On the TouchFLO 3D Home screen , slide to the Mail tab . 2. Swipe your finger up or down the screen to browse through received messages . Tap an e-mail message to open it . 3. Tap the To or Cc button to open the Select Contacts to Call screen . Using Phone Features 65 4. Select the check boxes of the sender and recipients who you want to join in 
 your conference call and then tap Conference . Tap here to close or expand the \u201c To \u201d or \u201c Cc \u201d recipient list . Select the check boxes of the participants . Tap Conference when you are ready to start the conference call . 5. Do steps 5 to 7 in \u201c To select participants and start a conference call \u201d to dial each participant and add each connected call to your conference call . 6. To manage your conference call , such as invite more participants , end the conference call , and more , see \u201c Managing the conference call \u201d in this chapter for details . 66 Using Phone Features 2.7 Turning the Phone Function On and Off In many countries , you are required by law to turn off the phone while on board an aircraft . To enable or disable the phone function 1. Tap Start > Settings , and then tap Communications . 2. Tap the OFF/ON slider at the right side of the Phone option to enable or disable the phone function . To enable or disable Airplane Mode Another way to turn off the ph
 one function is to switch your device to Airplane Mode . When you enable Airplane Mode , all wireless radios on your device are turned off , including the phone function , Bluetooth , and Wi-Fi . 1. Tap Start > Settings , and then tap Communications 2. Tap Airplane Mode to enable or disable Airplane Mode . When you disable Airplane Mode , the phone function is turned back on and the previous state of Bluetooth and Wi-Fi are restored . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/110TeleHTC_Manual_3_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/110TeleHTC_Manual_3_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/110TeleHTC_Manual_3_EN.txt.txt
new file mode 100644
index 0000000..9f9c452
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/110TeleHTC_Manual_3_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Chapter 3 TouchFLO\u2122 3D 3.1 Using the TouchFLO 3D Home Screen TouchFLO\u2122 3D gives you touch control over your Windows Mobile-powered device . TouchFLO 3D provides a Home screen that allows finger-touch access to the most widely-used functions , such as People , Messages , Mail , Music , and Internet . The TouchFLO 3D Home screen has the following tabs : Home Music Messages Calendar Stock Settings People Mail Internet Photos and Videos Weather Displaying the TouchFLO 3D Home screen in landscape view The TouchFLO 3D Home screen tabs are always shown at the bottom of the screen when the Home screen is in portrait view . When you slide out the hardware keyboard , the Home screen automatically changes to landscape view and temporarily hides the tabs . Touch the Expand button ( ) at the bottom left side of the Home screen to display the tabs . 68 TouchFLO\u2122 3D Expand button Switching between the tabs of the Home screen To switch between the tabs , you can do any of the following : \u2022 
 \u2022 Touch or tap the desired tab . Press and hold your finger on the active tab , and then slide your finger right or left over the tabs . Release when you have selected the desired tab . TouchFLO\u2122 3D 69 \u2022 Swipe left across the screen to move to the next tab , and swipe right across the screen to move to the previous tab . \u2022 Tip When using the hardware keyboard , press the right key to move to the next tab , and press the left key to move to the previous tab . While on any other TouchFLO 3D tab , press the END key to return to the Home tab . Customizing the Home screen tabs You can move or hide the tabs on the Home screen , except for the Home tab . The Settings tab cannot be disabled and hidden also from the Home screen . 1. On the Home screen , slide your finger to the Settings tab , and then touch Customize Tabs . 2. To move a tab , touch and hold the icon before the tab name , and then drag it up or down the tabs list . Release the icon when you reach the desired location 
 . 3. To hide a tab , clear the check box at the right of the tab name . 4. Touch Done . 70 TouchFLO\u2122 3D 3.2 Home The Home tab displays the current date , a flip-style clock , and an alarm icon that indicates whether the alarm is on or off . Missed calls and your upcoming calendar appointments are also shown on the Home tab . Open the Call History tab of the People screen and view the missed call information . Touch the clock to open the World Clock and change the date , time , and alarm clock settings . Swipe upward on the screen to display a smaller digital clock and view more appointments . To change back to the flip clock , swipe downward on the screen . Open the appointment details screen where you can view , edit , or delete the appointment . World Clock On the Home tab , tap the clock to open the World Clock tab ( ) and view the date and time . You can add , remove , or rearrange cities on the World Clock tab . Tip The World Clock tab can have up to 10 cities . TouchFLO\u2122 3
 D 71 Add another city . The date and time in selected cities . This is your local time . To change the local time to another city , select the button before the city name . Return to the Home tab . Remove or rearrange the cities , or adjust the local time . To change your local time , date and time zone 1. On the World Clock tab , touch Menu > Local Time Settings . 2. Adjust the time ( hour and minutes ) by tapping the up or down arrow buttons . 3. Touch Date , select the date , year , and month , and then touch Select . 4. Touch City . The Select Country screen then appears . 5. Flick your finger on the screen or use the Quick Scroll bar on the right of the screen to scroll through the country list . Touch the desired country to select it . 6. The Select City screen then appears . Flick your finger on the screen or use the Quick Scroll bar on the right of the screen to scroll through the city list . Touch the desired city to set your time zone . Tip You can also slowly scroll throu
 gh the country or city list by dragging your finger upward on the screen , or enter the country or city name in the search box . 7. Tap Done . Note When you synchronize your device with your computer , the time on your device is updated with the time on your computer . 72 TouchFLO\u2122 3D Alarm Clock On the Home tab , tap the clock , and then touch the Alarm Clock tab ( open the Alarm Clock screen . You can set up to three alarms on this tab . Touch an alarm time to open the Set Alarm screen and set the time and days when to sound the alarm clock . ) to Touch the OFF/ON slider to turn an alarm on or off . To set an alarm 1. On the Alarm Clock screen , touch an alarm time . 2. On the Set Alarm screen , adjust the alarm time ( hour and minutes ) , then tap the day of the week for the alarm . Tip Select multiple days by tapping each desired day . Tap a selected day to exclude it from the alarm . 3. Enter the alarm description , select the alarm ring tone , then choose whether you want th
 e device to vibrate when the alarm goes off . 4. Tap Done . 5. On the Alarm Clock screen , tap the OFF/ON slider to turn on the alarm . Note When an alarm goes off , you have the choice to Snooze or Dismiss the alarm . When you select Snooze , the alarm will sound again after 5 minutes . TouchFLO\u2122 3D 73 3.3 People On the People tab , you can add favorite contacts and quickly make calls or send text and e-mail messages to these contacts . From the People tab , you can also set up your own contact card , create new contacts , and easily view exchanged text , multimedia and e-mail messages with a contact . If contacts have their own Facebook account , you can view their Facebook updates and events right from the People tab . Note You need an active data or wireless network connection to the Internet to link to Facebook . See Chapter 8 for details on how to connect to the Internet . Setting up My Contact Card My Contact Card automatically opens the first time that you access the Peopl
 e tab . You can enter and save your contact information on My Contact Card so you can easily share and send it to your contacts . If you have a Facebook account , you can also sync your Facebook profile with My Contact Card so you can receive instant notifications everytime you or your Facebook friends change profiles or post events . 1. On the Home screen , slide your finger to the People tab . 2. When the My Contact Card screen appears , tap Next . Note Tap Skip if you prefer to set up My Contact Card later . You can open My Contact Card from the Settings tab of the TouchFLO 3D Home screen , or from the All tab of the People screen . See \u201c To view and search contacts on your device \u201d and \u201c Settings \u201d for details . 3. On the subsequent screen , choose how to add your photo to your contact card . You can use the Camera to take your photo , select a stored photo from your device or storage card , or if you have a Facebook account , login to your account and sync with your Fac
 ebook profile . Note When the Link Profile window opens , select or clear the Facebook information that you want to link , and then tap Done . 74 TouchFLO\u2122 3D 4. On the Detail Information screen , enter your information in the fields provided . You can scroll down the screen to view more fields . 5. When finished entering information , tap Next , and then tap Done on the subsequent screen to return to the People tab . Working with favorite contacts You need to have contacts already stored on your device before you can add them as favorite contacts on the People tab . If there are no existing contacts , create them first on your device . To create a new contact 1. On the People tab , touch All People . 2. The People screen then opens and displays the All tab ( create a new contact . ) . Touch New to 3. Enter the contact information in the fields provided and then tap Done or OK . Add the contact\u2019s photo . You can link to the contact\u2019s Facebook photo , select an existing photo i
 n Album , or open Camera and take a photo of the contact . Touch a field to start entering information . Add another phone number . Save the contact . TouchFLO\u2122 3D 75 Tips \u2022 To link with the contact\u2019s Facebook account , scroll down the New Contact Card screen and then tap Link in the Facebook section . For more information about Facebook , see \u201c Facebook updates and events \u201d in this chapter . \u2022 Tap More at the bottom of the New Contact Card screen to enter additional contact information from Windows Mobile\u2019s Contacts program . See \u201c Contacts \u201d in Chapter 12 for details . To add favorite contacts You can add up to 15 favorite contacts on the People tab . Note SIM contacts cannot be added as favorite contacts . 1. On the People tab , touch one of the plus ( + ) icons , or touch Add Favorite . 2. On the Select a Contact screen , touch the name of the desired contact . Tip To create a new contact , touch New on top of the list . 3. Select a phone number or e-mail addre
 ss to associate to the favorite contact . Note If there is no picture associated with the contact , the Album program opens and lets you take a photo of the contact or select an existing contact photo . For more information about Album , see Chapter 11. 4. To add another favorite contact , touch the small plus icon ( ) at the right side of the screen and repeat steps 2 and 3. 76 TouchFLO\u2122 3D To find a favorite contact Do any of the following : \u2022 \u2022 To flip through the contact pictures one at a time , swipe your finger upward or downward the screen . On the picture strip at the right side of the People tab : \u2022 \u2022 Touch the picture of the desired favorite contact ; or Press and hold on the currently selected contact picture at the right side of the screen , and then slide upward or downward the picture strip until the desired favorite contact is selected . Swiping upward/downward on the screen Tip Using the picture strip To remove the current favorite contact , touch Menu > Re
 move Favorite . TouchFLO\u2122 3D 77 To communicate with your favorite contact After selecting a favorite contact , do any of the following to communicate with that contact : \u2022 Touch the contact\u2019s name , phone number , or e-mail address below the contact picture to call or send an e-mail message . For more information about making a call or composing and sending an e-mail message , see Chapter 2 and Chapter 6. Touch an icon below the contact name to communicate with the contact . Call the contact\u2019s mobile phone . Call the contact\u2019s work/office phone . Call the contact\u2019s home phone . Compose and send an e-mail message . Compose and send a text message . \u2022 The icons shown depend on available contact information . For example , ( ) is shown if the contact information includes a home phone . \u2022 Touch the picture of the favorite contact on the center screen to open the Contact Details screen , and then choose how to communicate with the contact . See \u201c The Contact Details scr
 een \u201d in this chapter for details . 78 TouchFLO\u2122 3D The People screen Touch All People on the People tab to display the People screen where you can view and manage all your contacts . The People screen has these tabs : All , Favorites , Updates and Events , and Call History . All Call History Favorites Updates and Events Note The way to switch between tabs on the People screen is just the same as the way you switch tabs on the main Home screen . See \u201c Switching between the tabs of the Home screen \u201d earlier in this chapter . Functions Lists all contacts on your device and on the SIM card . This tab lets you create , delete , or edit a contact , or save a contact to your SIM card using SIM Manager . Touch a contact to open the Contact Details screen . You can also use this tab to open or update My Contact Card . Tip If you imported SIM contacts to your device and duplicated contacts are displayed , tap Menu > View > Contacts to filter and display only contacts stored on your d
 evice . Shows the favorite contacts that you have added to the People tab . You can also add favorite contacts through this tab . Displays notifications when there are updates that occurred in Facebook . This tab also shows events of your stored contacts . See \u201c Facebook updates and events \u201d in this chapter for details . Lists all your dialed numbers , and received and missed calls . See \u201c Using Call History \u201d in Chapter 2. Tab All Favorites Updates and Events Call History TouchFLO\u2122 3D 79 To view and search contacts on your device On the People screen , touch the All tab to view all your contacts . Swipe upward or downward on the screen to scroll through your contacts . You can also use the search box or the Quick Scroll bar to search for contacts on your device . Tap to send your contact information as an attachment in a multimedia message . Show or hide the search box . Use the search box to search contacts on your device . Create a new contact . Touch to edit or update 
 My Contact Card SIM card contact Touch to open the Contact Details screen . This icon appears when : \u2022 You receive a new text or e-mail message , or when you miss a call from the contact . Quick Scroll bar . Touch and hold , and then slide your finger downward . The Facebook status appears when you link the contact to his or her Facebook account . Add a new contact , delete contacts , filter \u2022 The contact has updated the contacts list , and more . Facebook profile or added Return to the People tab . events in Facebook . Note When you slide open the hardware keyboard and the screen switches to landscape view , the All tab automatically displays the search box . You can use the hardware keyboard to enter and search for a contact . 80 TouchFLO\u2122 3D The Contact Details screen When you touch a contact name in the All tab of the People screen or touch the picture of a favorite contact in the center screen of the People tab , the Contact Details screen opens . From this screen , you c
 an view the contact\u2019s details , exchange of messages and calls with the contact , and more . The Contact Details screen has these tabs : Details , Messages , Email , Updates and Events , and Call History . Details Email Call History Messages Updates and Events Note The way to switch between tabs on the Contact Details screen is just the same as the way you switch tabs on the main Home screen . See \u201c Switching between the tabs of the Home screen \u201d earlier in this chapter . Tab Details Functions Shows a contact\u2019s details . From this tab , you can call or send a text or e-mail message to the contact , edit or delete the contact information , or save the contact to your SIM card . Messages Displays exchanged text and multimedia messages with the contact . You can also reply with a text message directly on this tab . Email Displays all received e-mail messages from the contact . Updates and Events Displays notifications when the contact has updates on his or her Facebook account 
 . This tab also shows events of the contact . See \u201c Facebook updates and events \u201d in this chapter for details . Call History Lists all your incoming , outgoing , and missed calls to and from the contact . This tab also lets you call the contact . See \u201c Using Call History \u201d in Chapter 2 for details . TouchFLO\u2122 3D 81 Facebook updates and events If you and your stored contacts have Facebook accounts and you linked to Facebook when you were creating My Contact Card , you can easily check which of your contacts have updates that occurred in Facebook . The types of Facebook updates include changes in profile information and status as well as new photos added to the Facebook album . You can also easily check which contacts have events that occur on the current day . The types of events include the birthday and anniversary , if specified and stored on the contact details , as well as events that are created in Facebook . To check which contacts have Facebook updates and events You
  can check for Facebook updates and events of contacts from one convenient location , which is on the Updates and Events tab of the People screen . 1. On the Home screen , slide to the People tab and then tap All People . 2. Tap the Updates and Events tab ( ) . 3. If you haven\u2019t linked your stored contacts to Facebook , the Updates and Events tab will show a Link Profiles option . Tap this option , choose which contacts you want to link to Facebook , and then tap Next . On the Link Profiles window , select or clear the Facebook information that you want to link , and then tap Done . 4. In the Updates list , when you tap a contact who has only one type of Facebook update , Opera Mobile opens and brings you to your Facebook site where you can view details of the contact\u2019s Facebook update . When you tap a contact who has multiple Facebook updates , the Contact Details screen opens and displays the Updates and Events tab of this contact . See \u201c To check a single contact\u2019s Facebo
 ok updates and events \u201d for details . 5. In the Events list , tap a contact to send a greeting via a text message to the contact or view the contact details . If an event is a Facebook event , tap it to open Opera Mobile and view information about that event . 82 TouchFLO\u2122 3D Indicates that you have an invitation for an event in Facebook . Tap Me to open Opera Mobile and access your Facebook site so you can choose to accept or decline the invitation . Displays you and the contacts who have updates in Facebook . Lists contacts who have events on the current day . Notes \u2022 To manually refresh and display the latest Facebook updates , tap Menu > Update Now . \u2022 To enable the Facebook updates to be refreshed automatically , tap Menu > Data Connection Setting and then select the Download data automatically . The refresh is not instant . Your device checks from Facebook in a fixed time period . To check a single contact\u2019s Facebook updates and events To check only for a single cont
 act\u2019s Facebook updates and his or her events , access the Updates and Events tab from the Contact Details screen . 1. On the Home screen , slide to the People tab and then tap All People to open the People screen . 2. Tap a contact to open the Contact Details screen . 3. Tap the Updates and Events tab ( ) . 4. In the Updates list , tap a Facebook update to Opera Mobile and view details of that update . TouchFLO\u2122 3D 83 5. In the Events list , tap an event to send a greeting via a text message to the contact or view the contact details . If it is a Facebook event , tap it to access your Facebook site and view information about that event . To change your status in Facebook You can change your status in Facebook directly from your own contact card . 1. On the All tab of the People screen , tap Me to open your contact card . 2. Scroll to the bottom of the screen and then tap the button that displays your status . 3. Enter your new status and then tap OK . To log out from Facebook To
  log out from your Facebook account , open My Contact Card , scroll to the bottom of the screen , and then tap the message that says \u201c Tap to logout \u201d . 84 TouchFLO\u2122 3D To connect back to Facebook Do one of the following : \u2022 \u2022 \u2022 Open My Contact Card , scroll to the bottom of the screen , and then tap the message that says \u201c Login to my facebook \u201d . On the Updates and Events tab of the People screen , tap Link . On the Updates and Events tab of a single contact , tap Link Profiles . On the Facebook Login screen , enter the e-mail address and password that you use for your Facebook account . 3.4 Messages On the Messages tab , you can read text messages ( SMS ) as they arrive , and create a new text message . Touch the message to open the Messages tab of the Contact Details screen and view all received messages from the contact . Shows the number of unread text messages . Access all text messages in your inbox and other folders . Create a new text message . To flip thro
 ugh received text messages , swipe your finger upward/downward on the screen . Reply to the current message , delete the current message , and more . TouchFLO\u2122 3D 85 For more information about working with text messages , see \u201c Text Messages \u201d in Chapter 6. Making calls or sending e-mail directly from a text message While reading a text message on the Messages tab , you can directly call the sender of the text message . When you have opened a received text message , you can place a call to the phone number or send e-mail to the e-mail address that is contained in the message . For more information about directly making calls and sending e-mail while viewing text messages , see Chapter 6. 3.5 Mail The Mail tab is where you send and receive e-mail messages . Adding an Internet e-mail account On the Mail tab , use Mail Setup to easily add web-based e-mail accounts such as Yahoo ! Mail Plus , AOL , and more . 1. On the Home screen , slide to the Mail tab . 2. Touch New Account at 
 the bottom-right of the screen . The Mail Setup screen appears . 3. Enter your e-mail address and password , then tap Next . 4. Enter your name and the account display name , then tap Next . Mail Setup then searches for the e-mail provider settings on your device . If the settings are not found on your device , you will be prompted to enter the e-mail account settings manually . Obtain the e-mail account settings from your e-mail provider . 86 TouchFLO\u2122 3D 5. Tap the Automatic Send/Receive time , and then choose how frequent you want e-mail messages to be automatically sent and downloaded on your device . 6. Select or clear an advanced setting according to your preference , and then tap Done . For more information about specifying e-mail server settings , creating and sending e-mail messages , and customizing e-mail settings , see \u201c E-mail \u201d in Chapter 6. Adding an Outlook e-mail account You can also set up and add an Outlook e-mail account on the Mail tab and view the same e-
 mail messages that you receive in Outlook on your computer or on the Exchange Server . Before you add an Outlook e-mail account , obtain the Exchange Server name , domain name , user name , and password from your network administrator . 1. On the Home screen , slide to the Mail tab . 2. Touch New Account at the bottom-right of the screen . Note If you have previously added an e-mail account on the Mail tab , touch Menu > Account > New Account . 3. On the Mail Setup screen , enter your e-mail address and password , select Exchange Email ( Outlook ) , and then tap Next . 4. Enter the domain name and server address , select the This server requires an encrypted ( SSL ) connection option , if needed , and then tap Next . 5. Clear the data that you want to exclude from synchronization , and then tap Done . Your device synchronizes with the Exchange Server and downloads your Outlook e-mail messages . Note If you include all Outlook data , your Outlook contacts , appointments , and tasks w
 ill also synchronize with your device . For more information about working with company e-mail , see Chapter 7. TouchFLO\u2122 3D 87 Checking and reading e-mail messages The icon on the Mail tab and the respective icons of your e-mail accounts show the number of newly received or unread e-mail messages . Select an e-mail account and then swipe your finger up or down on the center screen to browse through received e-mail messages . \u2022 To browse through received e-mail messages , swipe up or down on the envelope image . \u2022 Touch the e-mail on the screen to open and read the message . Access all e-mail messages of the currently selected account . After selecting an e-mail account , touch this icon to create and send a new e-mail message . To select an e-mail account : \u2022 Touch an e-mail account icon to view the e-mail messages in that account . Or , \u2022 Touch and hold the icon of the e-mail account that is currently displayed , then slide upward or downward until the desired e-mail acc
 ount is selected . Create a new e-mail , add another e-mail account , send and receive e-mails , and more . For more information about creating and sending e-mail messages , see Chapters 6 and 7. Making calls directly from an e-mail message You can directly call the sender of the e-mail message that is currently shown on the Mail tab . When you have opened a received e-mail message , you can place a voice call or conference call to the other recipients of the e-mail . For more information about directly making voice calls and and conference calls while viewing e-mail messages , see Chapters 2 and 6. 88 TouchFLO\u2122 3D 3.6 Calendar The Calendar tab displays a monthly calendar and allows you to quickly create and view appointments . Indicates that you have appointments for that date . Today\u2019s date . Touch a date to view or create appointments for that date . Switch to Day view so you can view or create appointments for the current day . To go to the next or previous month , swipe you
 r finger upward/downward or touch the Up/Down arrow on the screen . From the menu , choose to create a new appointment for the current date , or go to the current date if it is not displayed on the screen . To create an appointment 1. On the Home screen , slide your finger to the Calendar tab . 2. If you are creating an appointment for the current date , touch Menu > New . Otherwise , touch another date on the calendar , and then touch New Appointment . 3. Enter a name for the appointment , then specify the location . 4. If there is a time frame for the appointment , set the start and end dates and times . If it is a special occasion such as a birthday or a whole-day event , set the All Day option to Yes . 5. When finished , tap OK to return to the Calendar tab . TouchFLO\u2122 3D 89 To view and manage your appointments On the Calendar tab , dates with appointments are indicated by a triangle ( the upper-right corner of the date box . 1. Touch a date on the calendar to switch to Day vi
 ew . The screen displays all your appointments and the weather information for that day . Note The weather information appears only if you have set your local time to a city on the World Clock tab , and if the appointment date is within the 5-day forecast on the Weather tab . See \u201c Home \u201d and \u201c Weather \u201d in this chapter for details . ) at 2. Touch an appointment to view its details . On the appointment details screen , touch Menu to edit , send , or delete the appointment . 3. Touch OK to return to Day view . Tip Your upcoming appointments also appear on the Home tab . To use Windows Mobile Calendar You can also use the Windows Mobile Calendar program to create , view , and send appointments . To open the Calendar program , tap Start > All Programs > Calendar . For more information about using Windows Mobile Calendar , refer to the \u201c Help & How-To \u201d topics on the Windows Mobile web site . 90 TouchFLO\u2122 3D 3.7 Internet On the Internet tab , you can open Opera Mobile\u2122 t
 o browse the web , search information , and add push web pages . This tab also lets you enable Push Internet to update the contents of your push web pages before you browse them . Enter the information you want to search on the web , and tap . Browse the web using Opera Mobile . Add a push web page . View your web favorites ( bookmarks ) . Swipe upward to scroll down the screen and access more web favorites , if any . Remove a push web page , adjust the auto-update time for your push web pages and bookmarks , and more . To add a push web page You can add push web pages on the Internet tab for quick access . 1. On the Home screen , slide your finger to the Internet tab . 2. Tap Add a new push page . On the New Push Page screen , enter the push web page title and address ( URL ) , and then select the auto update time under Frequency . Tip You can also select a bookmark , and add it as a push web page . 3. Tap Save . TouchFLO\u2122 3D 91 To automatically update your push web pages Automat
 ically update your push web pages to ensure that you browse the latest versions of your favorite web sites every time you access them from the Internet tab . The updated web sites are cached in the device memory so that they load faster in Opera Mobile . 1. On the Internet tab , touch Menu > Schedule Settings . 2. In the Day and Time section , select the day or days you want to auto update your push web pages , and then specify the update time . 3. Scroll down the screen to the Frequency section , then touch the update time below each of your push web pages to set the time interval for automatic updates . 4. Tap Done to return to the Internet tab . To configure Push Internet settings 1. On the Internet tab , touch Menu > Data Settings . 2. Touch the OFF/ON slider to enable or disable the following options : \u2022 Push Internet . Enable ( default ) or disable Push Internet . \u2022 Roaming . Enable this option if you want to update your push web pages and bookmarks even when you are roami
 ng . Enabling this option may incur additional data charges . This option is disabled by default . 92 TouchFLO\u2122 3D 3.8 Stock The Stock tab allows you to add , view , and analyze stocks and stock market movements using updated financial information . You can add up to 14 stock quotes or stock market indexes on the Stock tab . Note Downloading stock or stock market information may incur additional fees from your wireless service provider . Contact your service provider for details . To add a stock or stock market quote 1. On the Home screen , slide your finger to the Stock tab , and then touch Add Stock . Note If the Stock tab is not empty , touch Menu > Add Stock to add another stock quote or stock market index . 2. On the Add Stock screen , enter the company name , stock market index , or ticker symbol in the search box , and then tap . Search the company or stock market . Add the stock/stock market to the Stock tab . Enter the company/stock market name or ticker symbol . Return t
 o the Stock tab . Swipe upward to scroll down the screen and access more search results . 3. Tap the company name , stock market index , or ticker symbol to display it on the Stock tab . TouchFLO\u2122 3D 93 4. To add another stock quote or stock market index , touch Menu > Add Stock on the Stock tab , then repeat steps 2 to 4. To view stock quotes or stock market indexes The Stock tab lists all your stock quotes and stock market indexes including the current price or volume , and the change in price or volume from the previous trading day . The change in price or volume box indicates whether the stock or stock market has gained ( green ) or declined ( red ) . Open the Stock Details screen . Current trading price or volume Toggle between showing the change in points/price or percentage from the previous trading day . Touch the stock quote or stock market index name to view its intraday chart and other details . Touch to open the Stock Details screen . Shows the last time you updated th
 e information . To view more stock quotes or stock market indexes , swipe your finger upward on the screen . Add , delete , or rearrange the stocks and stock market indexes . Return to the Stock tab . Download the latest stock/ stock market information . 94 TouchFLO\u2122 3D To set the stock or stock market information download 1. On the Stock tab , touch Menu > Settings . 2. You can choose from the following download options : \u2022 Download data automatically . When selected , stock and/or stock market index data is automatically downloaded from the Yahoo ! Finance web site at the interval you specified . You can also update information every time an ActiveSync connection is established . Clear this option if you prefer to manually download stock and/or stock market index information . \u2022 Download data when roaming . Select this option only if you want to automatically download stock and/or stock market index information when roaming . This may incur additional data charges . TouchFLO
 \u2122 3D 95 3.9 Photos and Videos The Photos and Videos tab lets you flip through your photos and video clips and view them in full screen . From this tab , you can also open Camera and capture pictures and record videos . Capture a photo . Touch the photo or video to view or play it in full screen . Record a video . To flip through photos and videos , swipe upward/downward on the screen . Open Album . ( See \u201c Viewing Photos and Videos Using Album \u201d in Chapter 11 for details . ) View the photos as a slideshow . If a video is selected , touch Play . The Photos and Videos tab displays pictures and video files from the current favorite album . By default , the Camera Shots album is set as the favorite album . This album stores the pictures and videos you captured using the device camera . 96 TouchFLO\u2122 3D To set an album as a Favorite 1. On the Photos and Videos tab , touch Album . 2. Touch Albums and then select one of the listed favorite albums . 3. Tap Menu > Set Album as Favorit
 e and then tap OK on the confirmation screen . Tips \u2022 For more information about viewing in full screen , see \u201c Viewing photos \u201d and \u201c Playing videos \u201d in Chapter 11. The file formats that can be viewed on the Photos and Videos tab are the same as in Album . \u2022 To transfer media files from a computer and view them on the Photos and Videos tab , copy or synchronize them to a folder under \My Device ( the device memory ) or \Storage Card ( if a microSD card is installed ) . TouchFLO\u2122 3D 97 3.10 Music The Music tab allows you to visually browse through albums and music tracks and play music . The albums and music tracks that you see on the Music tab are from the Now Playing list of the Library . Go to the previous album or the previous track in the current album . Swipe upward/ downward on the screen to flip through albums or tracks in an album . Play or pause . Go to the next album or the next track in the current album . Set music playback to Repeat , turn Shuffle on or 
 off , and more . Access the Library . Note Your device searches for all supported music files from the following locations : Device : \My Music ( including all subfolders ) \My Documents ( including all subfolders ) Storage Card : \Storage Card ( including all subfolders ) For information about supported music file formats , see \u201c Specifications \u201d in the Appendix . To play music On the Music tab , flip through albums and tracks to find the music you want to start listening to , and then touch the Play icon . 98 TouchFLO\u2122 3D To move forward or backward in the current music track Touch and hold on the thin bar below the album art to display the progress bar : Drag your finger right or left on the progress bar to move forward or backward in the current track . To set repeat and shuffle modes Touch Menu > Repeat and choose whether to repeat the current track , repeat all tracks in the current album or playlist , or not to repeat . To turn shuffle mode on or off , touch Menu > Shuf
 fle and then select Shuffle On or Shuffle Off . The icons on the upper-right side of the Music tab show whether repeat and shuffle modes are enabled . Repeat [ One ] Repeat [ All ] Shuffle [ On ] To browse and play music in the Library The Library organizes music into categories , such as Now Playing , Artists , Albums , Genres , and All Songs . When you add more albums or music tracks to your device , go to the Library to locate and play the new albums or music tracks . 1. On the Music tab , touch Library to open the Library screen . 2. The tabs at the bottom of the Library screen correspond to the different categories . To browse music under a certain category , slide to the tab that corresponds to the desired category . 3. Keep tapping an item on a category tab ( for example , an artist ) until the music tracks display on the screen . 4. Tap a music track to play it . Note When you select a category in the Library such as Artists and play its music , the Now Playing list will be 
 replaced with the music tracks from that category . TouchFLO\u2122 3D 99 To create and use a playlist 1. Select the desired music on the Music tab or the Library to play it . 2. On the Music tab , touch Menu > Add to Playlist . 3. Touch Create new playlist , enter the Playlist Name and then touch OK . 4. Touch Library , slide to the Playlists tab , then touch a playlist to open it . 5. Touch the plus icon , or touch Menu > Add Songs , then slide to the All Songs tab . 6. Select the check boxes of the music tracks you want to add to the playlist , or touch Menu > Select All to choose all tracks . 7. Touch OK to add the tracks to the playlist . 8. When finished adding songs , touch Back to return to the Playlists tab . 9. On the Playlists tab , touch a playlist to open it . 10. Touch the first track in the playlist to play it on the Music tab . After each track , the next one in the playlist will be played . Note Playlists are not automatically updated when music files have been deleted 
 from the device memory or storage card . There are two types of playlists that can be shown on the Playlists tab : \u2022 \u2022 Custom playlists ( ) that are created on the Music tab . Windows Media� Player playlists ( ) that are in Windows Media� Player Mobile\u2019s Library ( which is synchronized with Windows Media� Player on your computer ) . These playlists cannot be edited . Note If a Windows Media� Player playlist contains a combination of music , video and image files , only music files in the playlist are synchronized to your device and other media types are filtered out . 100 TouchFLO\u2122 3D 3.11 Weather The Weather tab shows you the current weather as well as weather information for the next four days of the week . The Weather tab can display weather information of your local city and/or weather in up to ten cities . Shows the last weather update . To view the weather in other cities , swipe your finger upward/downward on the screen . Add or delete a location , change the temp
 erature to Celsius or Fahrenheit , and more . Download the latest weather information . Your device automatically connects to the Internet to download and display weather information . The current temperature , temperature range , type of weather ( sunny , cloudy , showers , or others ) , and the weather information for the next four days of the week are shown on the Weather tab . To add a city 1. On the Weather tab , touch Menu > Add Location . 2. The Select Country screen then appears . Flick your finger on the screen or use the Quick Scroll bar on the right of the screen to scroll through the country list . Touch the desired country to select it . TouchFLO\u2122 3D 101 3. The Select City screen then appears . Flick your finger on the screen or use the Quick Scroll bar on the right of the screen to scroll through the city list . Touch the desired city to select it . Tip You can also slowly scroll through the country or city list by dragging your finger upward on the screen , or enter
  the country or city name in the search box . To choose download options 1. On the Weather tab , touch Menu > Settings . 2. You can choose from the following options : \u2022 Download weather automatically . When this option is selected , weather information is automatically downloaded whenever you select the Weather tab , if the data has not been updated within the last three hours . Weather data is also downloaded every time an ActiveSync connection is established . Clear this option if you prefer to manually download weather information . \u2022 Download weather when roaming . Select this option only if you want to allow automatic download of weather data when roaming . This may incur additional data charges . 102 TouchFLO\u2122 3D 3.12 Settings The Settings tab gives you access to all the device settings . 1 2 3 4 5 6 7 Access all personal , system , and connection settings . 1 2 3 4 5 6 7 8 9 8 9 Adjust the Wi-Fi , and Bluetooth settings . Swipe your finger upward/downward on the screen
  to view more settings . Open ActiveSync and sync Outlook information . See Chapter 7 for details . Adjust the audio settings and sound profile . See \u201c Ringer settings \u201d in Chapter 13 for details . Change the Home tab wallpaper . Manage the communication features of your device . See \u201c Using Comm Manager \u201d in Chapter 13 for details . Move or hide the TouchFLO 3D tabs . See \u201c Customizing the Home Screen tabs \u201d earlier in this chapter . Set up My Contact Card . See \u201c Setting up My Contact Card \u201d in this chapter . Open Mail Setup and set up an Internet or Exchange Server e-mail account . See \u201c Mail \u201d earlier in this chapter . Enable or disable auto screen rotation or calibrate the device\u2019s G-Sensor . Display the TouchFLO 3D information . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/111TeleHTC_Manual_4_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/111TeleHTC_Manual_4_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/111TeleHTC_Manual_4_EN.txt.txt
new file mode 100644
index 0000000..1eafc71
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/111TeleHTC_Manual_4_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Chapter 4 Entering Text 4.1 The Hardware Keyboard Your device comes with a QWERTY keyboard , which is similar to a standard keyboard on a computer . To use the keyboard , slide the screen panel to the right to reveal the keyboard . The screen orientation automatically switches to landscape mode when the keyboard is opened . 104 Entering Text Adjusting the LCD screen angle After sliding out the hardware keyboard , you can adjust the angle of the device screen to minimize glare and reflection . 50� angle The maximum tilting angle of the screen is 50 degrees . Do not force the screen to tilt further than the maximum tilting angle as it will break the device . Before you retract the screen , please be sure to adjust the screen angle back to the original horizontal position . Using the hardware keyboard You can do the following with the hardware keyboard : \u2022 To type lowercase letters and numbers , press the keys using your thumbs or fingers . \u2022 To enter all uppercase letters , press
  twice , or press first and then press . To change back to all lowercase letters , press again . Entering Text 105 \u2022 To enter a single uppercase letter , press corresponding key . and then press the \u2022 To enter the symbol or punctuation mark , or launch the program that is printed at the upper right corner of a key , press , then press the corresponding key . \u2022 To open the Symbol Pad , press \u2022 Press \u2022 Press the arrow keys / / and then press / . to insert a hard return while typing . for directional movement . first and \u2022 To delete the character to the right of the blinking cursor , press then press . To delete the previous character , press . \u2022 Press \u2022 Press \u2022 Press to open the Messaging program . and then and then press to open your SMS/MMS messaging account . to open the XT9 menu . \u2022 Use the same way as you use the Control key of a computer keyboard . For example , when you are editing a Word document , you can do the following with this key : Press ... Ctrl + A
  Ctrl + B Ctrl + I Ctrl + U Ctrl + C To ... Select all Bold Italics Underline Copy Press ... Ctrl + X Ctrl + V Ctrl + Z Ctrl + Y Ctrl + Q To ... Cut Paste Undo Redo Exit Note Not all of the Ctrl key shortcuts may be supported in certain programs . 106 Entering Text 4.2 On-screen Input Methods When you start a program or select a field that requires text or numbers , the Input Panel icon is displayed on the menu bar . Tap the Input Selector arrow ( next to the Input Panel icon ) to open a menu where you can select a text input method and customize input options . After selecting a text input method , the corresponding Input Panel is displayed , which you can use to enter text . To show or hide the Input Panel , tap the Input Panel icon . Input Panel ( Full QWERTY ) Input Selector arrow Input Panel icon : Full QWERTY Phone Keypad Note Text input methods Select Other Input Methods to display more input methods . You can choose from Keyboard , which is the Windows Mobile on-screen keybo
 ard with QWERTY layout , Block Recognizer , Letter Recognizer , or Transcriber . Entering Text 107 4.3 Using the Full QWERTY The Full QWERTY is a full on-screen QWERTY keyboard layout similar to a computer keyboard . Note The keyboard layout may vary according to the region or country . \u2022 Tap to enter letters or symbols . \u2022 Tap and hold to enter the punctuation mark , symbol , or number at the top of a key . Tap to toggle between using Normal or Predictive mode . \u2022 Tap to enter an uppercase letter . \u2022 Tap twice to turn on Caps Lock . Displays the word candidate list in predictive mode . Tap a word to insert it into your text . Tap to add a word to the XT9 dictionary . Tap to delete the previous character . Tap to create a new line . Tap to open a keyboard layout that lets you easily enter numbers and symbols . See \u201c Using Numeric and Symbol Mode \u201d for details . To enter text using the Full QWERTY 1. Start a program that accepts text input , such as Word Mobile . 2. Tap t
 he Input Selector arrow , then tap Full QWERTY . 3. Start tapping keys on the keyboard to enter your text . 108 Entering Text As you tap the keys , the activated character displays on top of the key . Activated character Key when tapped . To learn how to enter text using Predictive mode , see \u201c Using Predictive Mode \u201d in this chapter . To enter accented characters 1. Tap and hold a key with accent options ( for example , vowels ) to display the accented characters bar . Accented characters bar Tapped key Slide your finger left or right on the screen to select an accented character . 2. Slide your finger left or right on the screen to select the accented character you want to enter . Entering Text 109 3. When you have selected the character , release your finger to enter it . Tip You can also use the Numeric and Symbol mode to enter accented characters . For more information , see \u201c Using Numeric and Symbol Mode \u201d in this chapter for details . 4.4 Using the Phone Keypad The P
 hone Keypad is an on-screen keypad that has a layout similar to mobile phone keypads , plus some additional keys . The Phone Keypad has large keys and enhanced input features such as predictive text input , which make it faster for you to enter text in your messages and documents . \u2022 Tap to enter letters or symbols . \u2022 Tap and hold to enter the punctuation mark , symbol , or number at the top of the key . Tap to toggle between using Normal or Predictive mode . \u2022 Tap to enter an uppercase letter . \u2022 Tap twice to turn on Caps Lock . Displays the word candidate list in Predictive mode . Tap a word to insert it into your text . Tap to add a word to the XT9 dictionary . Tap to delete the previous character . Tap to create a new line . Tap to open a keyboard layout that lets you easily enter numbers , symbols , and accented characters . See \u201c Using Numeric and Symbol Mode \u201d for details . 110 Entering Text To enter text using the Phone Keypad 1. Start a program that accepts tex
 t input , such as Word Mobile . 2. Tap the Input Selector arrow , then tap Phone Keypad . 3. When in multitap mode ( ) , enter a character by tapping a key until the desired character is displayed on the screen . As you tap the keys , the activated key displays just above the tapped key . \u2022 To enter the first letter that shows on a key , tap the key once . To enter the second letter , tap twice , and so on . \u2022 To enter a punctuation mark , symbol , or number that appears at the top of a key , tap and hold the key . Note You can also use the Predictive text input mode in Phone Keypad . For more information , see \u201c Using Predictive Mode \u201d in this chapter . 4.5 Using Predictive Mode XT9 is a predictive text input mode which displays a word candidate list as you enter the first few characters so you just select the word that you want . To enter text using Predictive mode 1. Make sure that the input mode is set to XT9 ( ) . 2. Start entering the first few letters of a word . As y
 ou enter each letter , predicted words appear in the word candidate bar . 3. Tap the desired word to enter . If the word is highlighted in the word candidate bar , you can also tap the Enter key or the Space bar to enter it . Note If there are several words to choose from , touch the left or right directional arrow keys below the input panel to navigate through the word candidate list . Entering Text 111 To add a word to the XT9 dictionary If the word that you have entered is not found in the dictionary , you can add it to the dictionary . 1. While in Predictive mode , type a letter and then tap word candidate list . 2. Type the word you want to add , and then tap key or the Space bar to add the word . on right side of the . You can also tap the Enter Cancel word addition . To customize Predictive mode settings 1. Tap the Input Selector arrow , then tap Touch Input Settings . 2. On the XT9 Settings tab , select or clear the following options according to your preferences : \u2022 Spell
  Correction . Lets you correct typographical errors by selecting from a list of possible words that reflect the characters of the keys you have tapped as well as characters of nearby keys . \u2022 Word Completion in XT9 mode . Lets you select from a list of possible words based on the keys that you have tapped and on the words available in the dictionary . \u2022 Word Completion in ABC mode ( Phone Keypad only ) . Lets you select from a list of possible combinations based on the characters that appear on the keys that you tapped . \u2022 Auto-Substitution . Automatically inserts an apostrophe when you type common contracted words ( for example , \u201c dont \u201d automatically becomes \u201c don\u2019t \u201d ) . 3. Tap OK . 112 Entering Text 4.6 Using Numeric and Symbol Mode Switch to Numeric and Symbol mode to easily enter numbers and common symbols such as parentheses , braces , currency signs , punctuation marks , and special characters . To switch to Numeric and Symbol mode 1. While using Full QWERTY
  or Phone Keypad , tap Tap a number or symbol to enter it . Go to the previous or next page of symbols . Tap to switch back to the letter keyboard layout . Phone Keypad Full QWERTY . 2. Locate and tap the desired number or symbol to enter it . To automatically switch to the letter keyboard layout By default , the keyboard in Numeric and Symbol mode is locked so you can continuously enter symbols and numbers . Do the following if you want to automatically switch to the letter keyboard layout after entering a symbol . 1. Tap the Input Selector arrow and then tap Touch Input Settings . 2. On the Options tab , clear Turn on symbol lock . 3. Tap OK . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/112TeleHTC_Manual_5_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/112TeleHTC_Manual_5_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/112TeleHTC_Manual_5_EN.txt.txt
new file mode 100644
index 0000000..a8ab596
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/112TeleHTC_Manual_5_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Chapter 5 Synchronizing Information 5.1 About Synchronization You can take information from your computer wherever you go by synchronizing it to your device . The following types of information can be synchronized between your computer and your device : \u2022 Microsoft� Office Outlook� information \u2014 Office Outlook e-mail , contacts , calendar , tasks , and notes \u2022 Notes created using Microsoft� Office OneNote� 2007 \u2022 Media \u2014 Pictures , music , and video \u2022 Favorites \u2014 Website addresses you save as Favorites in Pocket Internet Explorer on your device or in the Mobile Favorites folder of Internet Explorer on your computer . \u2022 Files \u2014 Documents and other files Tips \u2022 You can also synchronize Outlook e-mail , contacts , calendar , and tasks on your device with the Exchange Server at your work . For more information about setting up your device to synchronize with the Exchange Server , see Chapter 7. \u2022 Try to synchronize regularly in order to keep information up-to-d
 ate on both your device and your computer . Before you can synchronize , you need to install and set up first the synchronization software on your computer . For more information , see \u201c Setting Up Windows Mobile Device Center on Windows Vista� \u201d and \u201c Setting Up ActiveSync� on Windows� XP \u201d in this chapter . 114 Synchronizing Information After installing the synchronization software on your computer , connect the device to your computer using the USB sync cable . When the Connect to PC screen appears on your device , touch ActiveSync , and then touch Done . You can also synchronize information with your computer using Bluetooth . See \u201c Synchronizing via Bluetooth \u201d later in this chapter . Notes \u2022 ActiveSync or Windows Mobile Device Center only synchronizes data on your computer and device . If you have installed a storage card and you want to back up files from the storage card , use Windows Explorer on your computer to copy files from the storage card to your comp
 uter . \u2022 If you are sending your device for repair or performing a hard reset , data from your device will be lost . It is recommended that you sync your device to your computer to back up your files . To back up files from the storage card to your computer , use Windows Explorer to copy files from the storage card to the computer . 5.2 Setting Up Windows Mobile� Device Center on Windows Vista� Microsoft Windows Mobile� Device Center is the replacement for Microsoft� ActiveSync� on Windows Vista� . Some versions of Windows Vista� come with Windows Mobile Device Center already installed . If Windows Mobile Device Center is not available on your Windows Vista� , you can install it from the Getting Started Disc that came with your device . Setting up synchronization in Windows Mobile Device Center When you connect your device to your computer and start Windows Mobile Device Center for the first time , you are asked to create a Windows Mobile partnership with your device . To 
 create a partnership : 1. Connect your device to your computer . Windows Mobile Device Center configures itself , then opens . Synchronizing Information 115 2. On the license agreement screen , click Accept . 3. On the Windows Mobile Device Center\u2019s Home screen , click Set up your device . Note Choose Connect without setting up your device if you only want to transfer media files , check for updates , and explore your device but not synchronize Outlook information . 4. Select the items you want to synchronize , then click Next . 5. Enter a device name , then click Set Up . When you finish the setup wizard , Windows Mobile Device Center synchronizes your device automatically . Notice that Outlook e-mail messages and other information appear on your device after synchronization . 116 Synchronizing Information Using Windows Mobile Device Center To open Windows Mobile Device Center , click Start > All Programs > Windows Mobile Device Center on your Windows Vista computer . On Windows 
 Mobile Device Center , you can do the following : \u2022 Click Mobile Device Settings to change synchronization settings . \u2022 When you click Pictures , Music and Video > XX new pictures/video clips are available for import , a wizard guides you to tag and transfer photos from your device to the Photo Gallery on your computer . \u2022 Click Pictures , Music and Video > Add media to your device from Windows Media Player to synchronize music and video files using Windows Media� Player . For more information , see \u201c Using Windows Media� Player Mobile \u201d in Chapter 11. \u2022 Click File Management > Browse the contents of your device to view documents and files on your device . Note See Windows Mobile Device Center Help for more information . Synchronizing Information 117 5.3 Setting Up ActiveSync� on Windows� XP On a Windows� XP computer , you need to use Microsoft ActiveSync� 4.5 or later . If you do not have this software on your computer , you can install it from the Getting Started
  Disc that came with your device . Setting up synchronization in ActiveSync 1. Connect your device to your computer . The Synchronization Setup Wizard automatically starts and guides you to create a synchronization partnership . Click Next to proceed . 2. To synchronize your device with your computer , clear the Synchronize directly with a server running Microsoft Exchange check box , then click Next . 3. Select the information types that you want to synchronize , then click Next . 4. Select or clear the Allow wireless data connections check box according to your preference . 5. Click Finish . When you finish the wizard , ActiveSync synchronizes your device automatically . Notice that Outlook e-mail messages and other information appear on your device after synchronization . 118 Synchronizing Information 5.4 Synchronizing With Your Computer Connect and synchronize your device with your computer using the USB cable or Bluetooth connection . Starting and stopping synchronization You c
 an manually synchronize either from your device or computer . From your device \u2022 Tap Start > Settings , and then touch Sync Data ; or \u2022 Tap Start > ActiveSync , and then tap Sync . To end synchronization before it completes , tap Stop . Tip To delete a partnership with one computer completely , disconnect your device from that computer first . In ActiveSync on your device , tap Menu > Options , tap the computer name , then tap Delete . From Windows Mobile Device Center 1. Click Start > All Programs > Windows Mobile Device Center . 2. Click at the lower left of the Windows Mobile Device Center . . To end synchronization before it completes , click From ActiveSync on your computer When you connect your device to your computer , ActiveSync automatically opens on your computer and synchronizes . \u2022 To manually start synchronization , click . . \u2022 To end synchronization before it completes , click Synchronizing Information 119 Selecting information to synchronize You can select the
  information types and the amount of information to synchronize for each type either on your device or your computer . Follow the steps below to change synchronization settings on your device . Note Before changing synchronization settings on your device , disconnect it from your computer . 1. In ActiveSync on your device , tap Menu > Options . 2. Select the check box for any items you want to synchronize . If you cannot select a check box , you might have to clear the check box for the same information type elsewhere in the list . 3. To change synchronization settings for an information type , for instance , E-mail , select it and tap Settings . You can then set the download size limit , specify the time period of information to download , and more . Notes \u2022 Some information types such as Favorites , Files and Media cannot be selected in ActiveSync Options on your device . You can only select or clear these items from your computer\u2019s Windows Mobile Device Center or ActiveSync .
  \u2022 A computer can have sync partnerships with many different Windows Mobile powered devices , but a device can have sync partnerships with at most two computers . To ensure that your device synchronizes properly with both computers , set up the second computer using the same synchronization settings you used on the first computer . \u2022 Outlook e-mail can be synchronized with only one computer . 120 Synchronizing Information Troubleshooting sync connection problem In some cases , when the computer connects to the Internet or a local network , it may disconnect the connection with your device in favor of the Internet or network connection . If this happens , tap Start > Settings , and then tap All Settings . Tap Connections tab > USB to PC , then clear the Enable faster data synchronization check box . This makes your computer utilize a serial USB connection with your device . 5.5 Synchronizing via Bluetooth You can connect and synchronize your device with the computer using Bluetoo
 th . Note To connect and synchronize your device with a computer via Bluetooth , your computer must have a built-in Bluetooth or installed with a Bluetooth adapter or dongle . To synchronize with a computer via Bluetooth 1. Set up Windows Mobile Device Center or ActiveSync on your computer to synchronize through Bluetooth . See the program\u2019s Help for instructions . 2. Make sure that Bluetooth on both your device and the computer are turned on and set to visible mode . See \u201c Bluetooth Modes \u201d in Chapter 9 for details . If this is the first time you have connected to this computer via Bluetooth , you must first set up a Bluetooth partnership between your device and the computer . For more information about creating a Bluetooth partnership , see \u201c Bluetooth Partnerships \u201d in Chapter 9. 3. On your device , tap Start > ActiveSync . 4. Tap Menu > Connect via Bluetooth . Note To conserve battery power , turn off Bluetooth when not in use . Synchronizing Information 121 5.6 Synchr
 onizing Music and Video If you want to carry your music or other digital media along with you while you travel , set up Windows Media� Player on your computer to synchronize music and video with your device . Other than selecting the Media information type to be synchronized , all media synchronization settings must be set in Windows Media� Player . Before media can be synchronized , you must do the following : \u2022 Install Windows Media� Player Version 11 on your computer . ( Windows Media� Player 11 works only in Windows XP or later versions ) . \u2022 Connect your device to the computer with a USB cable . If your device is currently connected using Bluetooth , you must end that connection before media can be synchronized . \u2022 Set up a sync partnership between your device and your computer\u2019s Windows Media� Player . See \u201c Using Windows Media� Player Mobile \u201d in Chapter 11 for details . 122 Synchronizing Information 
\ No newline at end of file


[02/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/188un_CD_PV861.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/188un_CD_PV861.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/188un_CD_PV861.txt
new file mode 100644
index 0000000..171dab4
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/188un_CD_PV861.txt
@@ -0,0 +1,2 @@
+
+21 September 2000 Held at the Palais des Nations , Geneva , on Thursday , 21 September 2000 , at 10.40 a. m. President : Mr. Petko Draganov ( Bulgaria ) The PRESIDENT : I declare open the 861st plenary meeting of the Conference , the last for the 2000 session . I should like at the outset to extend a warm welcome , on behalf of us all , to the three new colleagues who have recently joined us as representatives of their Governments to the Conference , Ambassador Roberto Betancourt Ruales of Ecuador , Ambassador Rakesh Sood of India and Ambassador Mario Maioloini of Italy , and assure them of our full cooperation and support in the discharge of their functions . Allow me also to bid farewell to the representative of Ukraine , Ambassador Mykola Maimeskul , who , I understand , will soon be leaving his post to take up important responsibilities in his Government . Ambassador Maimeskul has represented his country in this Conference since August 1996 and we had the privilege to work under
  his presidency at a difficult time , when efforts were being made to widen the areas of agreement on the Conference 's programme of work . He discharged his functions with dedication and diplomatic skill . It is an especially moving occasion for me , because he was presiding over this Conference when I first appeared in it and you will also recall that it was under his presidency and skilful leadership of this Conference that the Conference took a decision on the establishment of the Ad Hoc Committee on fissile material . I am sure that you will all join me in extending to him and his family our best wishes for the future . I have on my list of speakers for today the representatives of Costa Rica , the Russian Federation and Algeria , who will be speaking on behalf of the Group of 21. I give the floor to the representative of Costa Rica , Ambassador Nora Ruiz de Angulo . Ms. RUIZ DE ANGULO ( Costa Rica ) ( translated from Spanish ) : Mr. President , allow me to convey greetings to 
 you , both on my own behalf and on that of the Government of Costa Rica , and to congratulate you on the way in which you have been presiding over this Conference . I also thank you for allowing me to speak before the plenary of the Conference on Disarmament , with a view to drawing the attention of distinguished representatives to Costa Rica 's renewed desire to become a permanent member of this Conference . We know of the difficult times through which the Conference is passing and of the various points of view on this subject and it is precisely for that reason that we wish to reiterate our aspiration and to commit ourselves to its attainment . Throughout its history Costa Rica has been distinguished for its pacifist tradition and its rejection of the use of force as a means to resolve conflicts . Since its beginnings as a republic , it has endeavoured to maintain relations of peace and friendship with neighbouring countries and to date it continues to participate in many internat
 ional activities and agreements aimed at reducing conventional armaments and weapons of mass destruction . Its most relevant decision in this regard was the abolition of the army in 1949. It thereby reaffirmed its commitment to peace and entrusted its safety to international law , rejecting force as a means of solving conflicts . From that time on , Costa Rica has entrusted its security to international rules and agreements and for that reason we wish to play an attentive and active role in addressing the international agenda on disarmament and international security . In 1958 Costa Rica submitted the first specific proposal on the limitation of conventional weapons and the prohibition of nuclear weapons in the Organization of American States ( OAS ) . Subsequently we ratified with great enthusiasm the Treaty for the Prohibition of Nuclear Weapons in Latin America and the Caribbean . Costa Rica firmly supports general and complete disarmament . Ever since the San Francisco Conferenc
 e , at which Costa Rica was a participant , we have continued to place our firm trust in the United Nations and we are strongly committed to its mandate to safeguard international peace and security . We shall therefore continue to support the work of the First Committee and the Commission on Disarmament by preparing , co-sponsoring and considering proposals conducive to international disarmament . To this end we advocate the creation of nuclear-free zones , complete nuclear disarmament , the holding of the fourth special session of the General Assembly on disarmament and the expansion of the United Nations Register of Conventional Arms . We consider there should be a ban on the transfer of materiel and military personnel and the provision of financial or logistic support to States whose military or paramilitary units , or whose security forces are responsible for violations of human rights or contribute to such violations . At the first preparatory meeting of the 2001 United Nation
 s Conference on the Illicit Trade in Small Arms and Light Weapons , held on 3 March 2000 , the Costa Rican Government officially presented a working document on an international code of conduct on arms transfers . The Preparatory Committee elected our country to the vice-chairmanship of the United Nations Conference , to be held in 2001. During its participation on two occasions as a non-permanent member of the Security Council , most recently in 1998 , Cost Rica focused its efforts on preventive diplomacy , the resolution of conflicts , humanitarian assistance and the protection of human rights in war zones , both during and after hostilities . Within the framework of OAS Costa Rica has proposed a set of practical guidelines for the attainment of disarmament as a dividend for peace . These guidelines are designed to ensure a genuine reduction in conventional weapons , so that more resources can be devoted to development . The agreements ratified by Costa Rica include the Treaty on 
 the Non-Proliferation of Nuclear Weapons ; the Treaty on the Prohibition of the Emplacement of Nuclear Weapons and Other Weapons of Mass Destruction on the Seabed and the Ocean Floor and in the Subsoil Thereof ; the Convention on the Prohibition of the Development , Production and Stockpiling of Bacteriological ( Biological ) and Toxin Weapons and on Their Destruction ; the Convention on the Prohibition of Military or Any Other Hostile Use of Environmental Modification Techniques ; the Convention on Prohibitions or Restrictions on the Use of Certain Conventional Weapons Which May Be Deemed to Be Excessively Injurious or to Have Indiscriminate Effects and its three protocols ; the Convention on the Prohibition of the Development , Production , Stockpiling and Use of Chemical Weapons and on Their Destruction ; the Comprehensive Nuclear Test-Ban Treaty ; the Convention on the Prohibition of the Use , Stockpiling , Production and Transfer of Anti-Personnel Mines and Their Destruction ; 
 and the Inter-American Convention against the Manufacturing of and Trafficking in Firearms , Ammunition , Explosives and Other Related Materials . Costa Rica attaches great importance to multilateral forums and focuses its activities in the area of security on the strengthening of collective security agreements , of both regional and international scope , and on the promotion of the peaceful solution of conflicts and disarmament as a basic principle of its foreign policy . Costa Rica 's achievements in the national and international arena are attributable to our sound decision , made more than 50 years ago , to abolish our army as this has enabled us to devote more resources to social investment , to renounce weapons and to promote democracy and development . Given that Costa Rica 's efforts to promote the security and well-being of the country 's society are largely linked to the development of international law and the effectiveness of the protection which it provides , we deem it
  essential that we are properly represented in these forums . Where international law prevails , the risk of war are reduced and we therefore consider it necessary to continue preparing effective verification mechanisms that guarantee that there is no use , threat of use or manufacture of weapons of mass destruction . In the light of all that I have said , Costa Rica reiterates once more its request to be admitted to the Conference on Disarmament . Over the last 50 years Costa Rica has not spent any money on an army and has devoted its budget to education , preventive health-care and infrastructure investment . Our national experience has taught us that to decide not to invest in weapons is the best way forward for those countries that are really committed to the well-being of their peoples . We believe that the promotion of peace and security , both internationally and on the domestic front , is contingent upon a determined policy to reduce military expenditure and to strengthen ci
 vil power . It stands to reason that disarmament , demilitarization and the reduction of military expenditure are essential for the welfare of mankind and the promotion of a culture of peace . The PRESIDENT : I thank the representative of Costa Rica for her statement and for the kind words addressed to the Chair . I now give the floor to the representative of the Russian Federation , Ambassador Vasily Sidorov . Mr. SIDOROV ( Russian Federation ) ( translated from Russian ) : First , may I thank you for giving me the floor and congratulate you on your assumption of the presidency of the Conference . I wish you every success and assure you of my delegation 's willingness to cooperate in the most constructive manner . I would also like to welcome those new colleagues who have recently joined us in our work in the Conference . At the Millennium Summit , held recently in New York , attention was given to an issue of particular importance to the modern age : what system of international r
 elations will prevail over our lives and activities in the twenty-first century ? We believe that the new age should be one of equal security and a just peace , as stated by President Putin of Russia in his speech at the Summit : \u201c The new age for the United Nations must continue well into a millennium of global stability . It must go down in history as a period of real disarmament . It has already proved possible to create an efficient disarmament machinery . That machinery comprises the 1972 ABM Treaty , non-proliferation regimes governing weapons of mass destruction and their means of delivery and dozens of highly important agreements on the reduction and limitation of various weapons . \u201d I have already had ample occasion in this auditorium to state the abiding significance of the ABM Treaty for strategic stability and international security . The issue has recently been raised again , more than once , in plenary meetings of the Conference on Disarmament and points have been 
 made which require clarification on our part . We are firmly convinced that the ABM Treaty remains the cornerstone for strategic stability and that it continues to serve as a basis for ensuring further reductions in strategic offensive arms . We believe that this linkage will be further strengthened in the foreseeable future . Incidentally , it was the United States which was the first to raise the issue of a connection between agreements to reduce and limit strategic offensive arms and the ABM Treaty . On 9 May 1972 , during the final stage of preparations for the ABM Treaty , Ambassador Smith of the United States of America made a statement to the effect that the United States reserved the right to withdraw from the Treaty unless the START agreement was concluded within five years . The significance and historic role of the ABM Treaty are not confined merely to the sphere of Russian-United States relations . It underpins the entire modern system of arms control agreements and the 
 collapse of the ABM Treaty would therefore disrupt the whole range of disarmament agreements created over the last 30 years . There is an increasing threat of the erosion of the regimes of non-proliferation of weapons of mass destruction and their means of delivery . The creation and deployment of the United States national missile defence system would give powerful impetus to the proliferation of missiles and missile technology . The positions of many countries on nuclear disarmament would shift and the Treaty on the Non-Proliferation of Nuclear Weapons would fall victim to these developments . Those who oppose the idea of preserving the ABM Treaty in its present shape often say that nothing would happen if it were to be slightly amended and adjusted to current realities , particularly since corrections have already been made to it before . Let us look more closely at this argument . It is true that the 1972 ABM Treaty does make allowance for the possibility of changing its provisi
 ons and the parties to the Treaty have made use of that possibility in the past . The intent of those changes was to strengthen the Treaty 's regime and to enhance its \u201c restraining \u201d elements . Today , however , it is suggested that we amend this instrument so as to accommodate the deployment of a \u201c limited \u201d anti-missile defence system for a national territory . This is contrary to the core provisions of article 1 , which constitute the very essence of the ABM Treaty . In other words , what is being suggested is that the ABM Treaty be changed from one which prohibits the creation of national missile defence systems to one which actually provides for the creation of such systems , its very opposite . Such a change in the Treaty would set a precedent in disarmament practice whereby the introduction of additions or amendments to an agreement limiting and restraining the development of an advanced weapon could turn it into an agreement which actually permits military build-up 
 . We cannot accept this sort of logic . The announcement by President Clinton of the United States that he would not enter into any commitment to deploy a national missile defence system is seen in Russia as a well-considered and responsible decision . As President Putin stresses , however : \u201c This does mean that Washington and Moscow now share completely identical approaches to the issue of ABM \u201d . We cannot turn a blind eye to the fact that programmes are still under way to set up a national missile defence system in the United States of America . No one has repealed the National Missile Defence Act adopted in the United States in 1999. According to our analysis it is difficult to conceive of a United States national missile defence system which would not impair Russia 's deterrent potential and yet would still fall within the limits of the ABM Treaty . Among the implications of creating any United States national missile defence system would be the disruption of the prevailin
 g strategic stability and the devaluation - to some extent or other - of Russia 's strategic capability . It would also ensure the unilateral military superiority of the United States of America . In these circumstances , assurances by the United States that it is committed to the ABM Treaty , that its national missile defence system is supposedly not directed against Russia and that it stands ready to pursue \u201c trade-off \u201d cooperation with Russia in the area of ABM cannot allay our fears . We are not shirking dialogue with the United States of America . This does not , however , imply any consent whatsoever on our part to adapting the ABM Treaty to accommodate the United States national missile defence system . Our position is dictated by the best interests of strengthening peace and international stability . We do not intend to change our approach . Our message to our United States partners is clear and plain : we will not participate in the destruction of this fundamental inst
 rument - for this , effectively , is what is at stake . Russia is ready to continue active joint efforts both with the United States of America and with other interested States to ensure international security , including through prompt progress towards agreements on further reducing nuclear arsenals , improving the regimes for nuclear and missile non-proliferation and strengthening strategic stability by political means . We recently ratified the START II Treaty , the 1997 New York package of START and ABM-related agreements and the Comprehensive Nuclear Test-Ban Treaty . We are now awaiting reciprocal steps by the United States of America . We advocate the immediate start of negotiations on START III . During the Okinawa summit , President Putin submitted to President Clinton detailed proposals regarding the main areas of the START III negotiations . These are based on our belief that the conclusion of a START III Treaty is only possible if the ABM Treaty remains inviolate . Let u
 s now try to respond to the arguments of those advocates of amending the ABM Treaty who invoke the changed strategic situation and the increased missile threats . We consider that references to such a threat have no real foundation . The deployment of a United States national missile defence system is motivated by the necessity to counter the strategic ballistic missiles of certain countries . However , the proliferation of such ballistic missiles is not expected to occur either today or in the foreseeable future . The changes in the strategic situation cited by the proponents of amendments to the ABM Treaty are in no way comparable to the changes in the strategic situation that could occur if the Treaty were to be undermined and the disarmament process destroyed . We are convinced that the issue of missile proliferation can and must be dealt with without disrupting the ABM Treaty . Instead of the military solution to the issue , we suggest a constructive approach , through politica
 l and diplomatic measures . We are ready to engage in the broadest possible consultations on the whole range of missile non-proliferation issues with all interested countries . Furthermore , as you are aware , initial results are already available from work with those countries mentioned by the United States as posing a missile threat . We are satisfied that the United States has started to give positive consideration to Russia 's ideas concerning the creation of a global missile and missile technology non-proliferation control system . It is significant that the joint Russian-United States statements on the outcome of the recent meetings between the presidents of the two countries in Okinawa and in New York registered their readiness for joint action to resolve the issue of missile proliferation , including the implementation of Russia 's proposals for a global control system . The role of the ABM Treaty goes well beyond arrangements among the narrow circle of States to which it is
  formally confined . This was clearly demonstrated by the results of the vote at the fifty-fourth session of the General Assembly on the resolution on the preservation of and compliance with the ABM Treaty , which was put forward by Russia jointly with Belarus and China . We see these results as endorsement of our efforts to enhance the effectiveness and authority of that Treaty . Any country 's attempt to strengthen its own security at the expense of that of others is doomed to failure , as history has shown . The collapse of the ABM Treaty and the consequent collapse of all the other nuclear disarmament treaties based on it would undoubtedly cause many States to reconsider their priorities and methods for ensuring security . In this regard , the resolution adopted in 1999 by the General Assembly sends an unmistakably clear message to those seeking to disrupt the ABM Treaty . We plan to submit a draft resolution to the General Assembly at its current session in support for the ABM 
 Treaty . The text of the draft , which is virtually identical to that of last year 's resolution , was circulated a few days ago to all delegations here in Geneva . The draft is not confrontational . It is based on the language of the Treaty itself and on the joint statements on the issue by the Presidents of Russia and the United States of America . It is not directed against any country , nor does it prejudice anyone 's interests . Its purpose is clear and entirely constructive : to ensure that , through rigorous and full compliance with the ABM Treaty , the Treaty is preserved and strengthened . Adoption by the General Assembly at its current session of the draft resolution on the preservation of the ABM Treaty would send a new signal to the international community on the importance of ensuring the viability of this Treaty . We expect our draft to receive broad support from the international community . The task of preserving the ABM Treaty under current conditions is closely rel
 ated to the prevention of an arms race in outer space , another issue that has been entrusted to our Conference . The ABM Treaty prohibits the development , testing or deployment in outer space of missile defence components . Application of its regime will keep outer space free of an entire class of space weapons - anti-missile weapons . We are firmly convinced that the preservation of this agreement will constitute a major obstacle to any possible escalation of the arms race , both here on Earth and in outer space . We share the view of those delegations who consider that the elaboration at this early stage of specific measures to prevent an arms race in outer space will help obviate the enormous costs involved in disarming space in the future . In this context , I would like to draw the attention of the participants here in the Conference on Disarmament to the proposal made by President Putin at the Millennium Summit , that an international conference on the prevention of demilita
 rization of outer space should be convened in Moscow in spring 2001 under the auspices of the United Nations . The Russian Federation supports the re-establishment in the Conference on Disarmament of an ad hoc committee on the prevention of an arms race in outer space with the mandate to elaborate specific practical arrangements which will prevent near-Earth space from being transformed into a new arena for confrontation between the great Powers . As we see it , the international community should set as one of its priorities the earliest possible elaboration of an international legal regime prohibiting the introduction in outer space of other weapons besides weapons of mass destruction , primarily strike weapons . At the same time , it is our firm view that the work of the Conference on Disarmament on outer space issues should not obstruct the peaceful exploration of outer space . I should like to reaffirm that we look favourably on the groundwork being carried out by Canada on the 
 non-weaponization of outer space , the French ideas for a notification regime for launches of space objects and ballistic missiles and other proposals by participants in the Conference which could serve as a basis for substantive discussion . In concluding my statement , I would like to cite the words of President Putin at the Millennium Summit , \u201c I am convinced that by building upon a just world order and strategic stability , we shall ensure sustainable development for civilization . The Russia of today is , as never before , open , responsible and ready for cooperation on the basis of equal partnership . \u201d The PRESIDENT : I thank the representative of the Russian Federation for his statement and for the kind words addressed to the Chair . I now give the floor to the representative of Algeria . Mr. BENFREHA ( Algeria ) ( translated from French ) : I thank you , Mr. President , for all the efforts you are making and for the way in which you are conducting our work . I also mus
 t pay particular tribute to Mr. Abdelkader Bensmail , Deputy Secretary-General of the Conference , for the work he has done in the Conference . I have the honour , on behalf of the Group of 21 , to make the following statement , which I shall deliver in English : ( continued in English ) As the 2000 session of the Conference on Disarmament draws to a close , the Group of 21 regrets that the Conference on Disarmament , as the single multilateral negotiating forum on disarmament , has once again been unable to agree on a programme of work during its current session . This failure is primarily due to the continued inflexibility of some of the nuclear-weapon States regarding negotiations on nuclear disarmament and on measures to prevent an arms race in outer space . The Group of 21 emphasizes that nuclear disarmament remains the highest priority for the Conference on Disarmament and stresses the necessity of establishing in the Conference an ad hoc committee on nuclear disarmament under
  agenda item 1. In this regard , the Group invites the Conference 's attention to its proposals contained in CD/1570 , CD/1571 and CD/1549 and to the statement made by the coordinator of the Group on 27 January 2000. The Group emphasizes that outer space is the common heritage of humankind . In this regard , the prevention of an arms race in outer space has assumed greater urgency because of legitimate concerns that existing legal instruments are inadequate to deter imminent attempts aimed at the further militarization of outer space and its possible weaponization . In accordance with resolution 54/53 of the United Nations General Assembly , the Group emphasizes the urgent need for commencement of substantive work in the Conference on Disarmament on the prevention of an arms race in outer space . In this connection , the Group calls upon the States parties to the ABM Treaty to comply fully with its provisions . The Group of 21 reaffirms its flexibility and readiness to continue to c
 ontribute constructively and extends its full support to the efforts of the President of the Conference . The Group of 21 urges other groups to display matching flexibility so that substantive work can commence early next year within the framework of a programme of work which reflects the priorities and interests of all delegations . The PRESIDENT : I thank the representative of Algeria for his statement and the kind words addressed to the Chair . That concludes my list of speakers for today . Does any other delegation wish to take the floor at this point ? I give the floor to Ambassador Hu of China . Mr. HU XIAODI ( China ) ( translated from Chinese ) : Mr. President , I have listened attentively to the previous speakers . Last week I took the floor to convey , yet again , the views of the Chinese delegation on national missile defence and the work programme of the Conference on Disarmament . I would like to take this opportunity to present further views on these two issues . First
  , with regard to national missile defence , the reason advanced for developing and employing such systems is defence against the missile threat posed by certain so-called \u201c countries of concern \u201d . It is manifest to all , however , that those countries , given their military capacity and technology and their overall national strength , simply are not in a position to pose any military threat in the foreseeable future to the militarily most powerful country in the world . They are even less likely to pose any threat of missile-borne weapons of mass destruction . Still more important is the lack of motive . No small country would ever attack or threaten to attack the militarily most powerful country in the world with missile-borne weapons of mass destruction , unless it had decided to commit suicide . For that reason , attributing the need to develop and deploy a national missile defence system to the missile threat posed by so-called \u201c countries of concern \u201d is far from conv
 incing . It is only a clumsy pretext . International arms control , disarmament and security treaties concluded in the past are not necessarily unamendable in the light of new circumstances . Any amendment ought , however , to promote the purpose and objective of the treaty concerned , namely , international peace and security . If the effect of an amendment is to undermine the treaty 's purpose and objective and to jeopardize international peace and security it must be rejected and opposed . The attempt to amend the ABM Treaty to allow the deployment of national missile defence systems is precisely such a case . It is an incontestable fact that the country possessing the world 's most sophisticated space technology is busy pursuing its plan to develop space weapons . The memory of the \u201c Star Wars \u201d programme of the 1980s remains fresh in our mind . The long-range plan to develop a space military strategy , drawn up in April 1998 , is also no secret . It explicitly states that m
 ilitary space capabilities will become a major factor in ensuring that country 's national security and implementing its military strategies and that the primary task of its space force in the twenty-first century is to seek superiority in space . For that purpose , the space command of that country has promoted such operational concepts as securing control of space and global engagement . This is the background against which the development of space weapons has been stepped up , with the national missile defence programme as part of the overall effort . Space systems are an important component of the national missile defence programme . Space will also become the battlefield for national missile defence . The national missile defence system currently being developed is only the first stage in the overall national missile defence plan . More advanced national missile defence systems will follow ; even more weapon systems will be deployed in outer space . The danger of the weaponizat
 ion of outer space and an arms race in outer space is therefore very real . It is essential for the international community to negotiate now an international treaty to prevent the weaponization of outer space and an arms race in outer space . With regard to the work programme of the Conference on Disarmament , to date the Conference has been unable to carry out any substantive work for two consecutive years . This is a matter of profound concern to China and it earnestly hopes that the Conference will be able to break the impasse as soon as possible . As everyone can see , the impasse is caused by the different parties taking different items of the Conference 's agenda as their top priority . Is there any way out of this impasse ? In my opinion , the answer is yes . What we need to do is to take the concerns of all sides into consideration in a comprehensive and balanced way and set up ad hoc committees with negotiating mandates on the priority items of all sides . That is the only 
 approach that will bring a solution that is genuine and fair without favouring any side , and therefore realistic and workable . We will never reach consensus on the work programme and break the impasse if any one delegation insists on negotiating only on its own priority items and will not permit negotiations on the priority items of other parties . Regrettably , some delegations are persisting in this , which has thus far kept the Conference on Disarmament from establishing any working mechanism , including ad hoc committees , on nuclear disarmament and outer space and prevented it from carrying out any substantive work . China advocates the establishment of an ad hoc committee on outer space , to negotiate a legal instrument to prevent the weaponization of outer space and an arms race in outer space . I have already had occasion , last week and many times before , to elaborate on our reasons for this . China 's position on nuclear disarmament is consistent . We support the positi
 on of the Group of 21 with regard to the establishment of an ad hoc committee to conduct negotiations on nuclear disarmament . Although negotiations on FMCT are not a priority item for us , China does not oppose such negotiations . However , in view of certain negative developments , including the setback of CTBT and , in particular , the ongoing national missile defence programme , any formula would be unacceptable to us if it led only to negotiations on FMCT , without also leading to negotiations on outer space . It is clear that there are two possible approaches . One is that advocated by China , namely , to launch negotiations on the priority items of all parties . The other approach is to negotiate only on the priority of one country and not to permit negotiations on the priorities of others . China 's approach , if acceptable to all , could certainly lead to consensus on the Conference 's work programme . By contrast , the other approach , if accepted by all , could never lead
  the Conference out of its impasse . It is the hope of the Chinese delegation that in the future all sides will treat each other equally and deal with issues in a just and fair way , rather than attending only to their own interests at the expense of the interests of the majority of other countries . In conclusion , Mr. President , the Chinese delegation would like to convey its appreciation for your continued efforts to find a good solution to the problem of the work programme and for your guidance in preparing the annual report of the Conference on Disarmament and the draft General Assembly resolution . We are also grateful to the distinguished Secretary-General and Deputy Secretary-General of the Conference and to all the staff of the secretariat and of the Conference Support Branch for the enormous help they have given us in our work . Mr. MAIMESKUL ( Ukraine ) ( translated from French ) : Mr. President , you still have New York and the General Assembly ahead of you , so let me 
 take this opportunity to wish you every success in your mission . I would like in particular to thank you for your kind words to me . You will recall that , at the height of Ukraine 's presidency two years ago , it was my privilege to welcome you to Geneva . Two years ago , we were able to overcome the difficulties , uncertainties and , on occasion , opposition which faced us and we resolved to move forward . Notwithstanding the difficult times which the Conference is currently experiencing , I believe in its potential and its future . As was said one and a half years ago by Ambassador Jo�lle Bourgois of France , the very existence of the Conference on Disarmament is a confidence-building measure which nothing can ever replace . All we need now is a result . Before concluding , Sir , I would like to thank all my colleagues for their cooperation and to bid them all farewell for now , but not for ever . I thank the Secretary-General of the Conference , Mr. Petrovsky . I thank you , M
 r. Bensmail , for having so generously shared your unique experience with the Ukrainian delegation , particularly during our tenure as President . I thank the whole team of the Conference secretariat for their readiness to help , their kind consideration and their counsel . The PRESIDENT : I thank the representative of Ukraine for the kind words addressed to the Chair and for his statement . I now give the floor to the Ambassador of Ecuador . Mr. BETANCOURT RUALES ( Ecuador ) ( translated from Spanish ) : Thank you , Mr. President . For me it is a great honour to speak to the distinguished members of this important forum , as I begin my term of office in the area of disarmament as head of the delegation of Ecuador . I thank you for your kind words of welcome , which commit me not only to the primary objectives of this Conference but also to continued cooperation in the essential work that all Governments are carrying out to maintain international peace and security . For these reaso
 ns , Sir , I must express my gratitude for all the efforts made by your predecessors and , in particular , my full support for your own efforts as President . In my view , the Conference on Disarmament has as its purpose the coordination of all political aspirations on multilateral issues related to peace , with particular emphasis on disarmament , and progress towards a new , positive climate in international security . The uncertainties and the challenges that this Conference has come up against in recent years may eventually be dispelled if all members heed the call of the international community by cooperating more closely and striving , for example , for consensus on a necessary programme of work for the Conference . Ecuador associates itself with those countries which have stated that this Conference must not shirk its responsibility any longer . It must begin the new millennium with an agenda that will safeguard the process of nuclear disarmament and nuclear non-proliferation
  , an agenda that will be very clear and inspire hope in all humankind . Ecuador believes that nuclear disarmament is a priority of the international community ; at the same time disarmament is one of the greatest challenges of our times , and one that we must overcome as soon as possible . The PRESIDENT : I thank the representative of Ecuador for his statement and the kind words addressed to the Chair . I now give the floor to the representative of India . Mr. SOOD ( India ) : Mr. President , I have sought the floor to thank you for your warm words of welcome to me this morning . I would also like to convey my appreciation for the many friendly faces that I see around this room as I return to this chamber after an absence of almost 11 years . I look forward very much to working with all my colleagues here and I am quite sure that we should be able to move things forward in the months to come , as we have under your presidency this month . I would like to add that we are aware of th
 e fact that , for the last couple of years , we have not seen active negotiations taking place in the Conference on Disarmament , but nonetheless , the fact that my Government has chosen this moment to appoint me here as head of the Indian delegation to the Conference on Disarmament is a reflection of India 's deep commitment to multilateral negotiations and of its awareness of the importance of multilateral negotiations in disarmament and the unique position that this particular institution has in this field . The PRESIDENT : I thank the representative of India for his statement and the kind words addressed to the Chair . I now give the floor to the representative of the United States of America . Ambassador Grey , you have the floor . Mr. GREY ( United States of America ) : I regret that the distinguished representative of China has seen fit again to misrepresent the plans and intentions of the United States . In my view , these misrepresentations become less persuasive with each 
 repetition . When I spoke on 31 August and 14 September , I addressed these questions in considerable depth . It would be a disservice to the Conference and disrespectful to those here present for me to repeat the same comprehensive answers I have already given , so I shall refrain from doing so today . But I will note that the way to get the Conference on Disarmament back to work is to move along the lines suggested by three past presidents , the ambassadors of Algeria , Belgium and Brazil . My delegation is prepared to proceed in this direction and , I would remind you , so too are the vast majority of the Conference 's members . The PRESIDENT : That concludes the list of speakers that I have for today . Does any other delegation wish to take the floor ? That does not seem to be the case . I will then invite you to formalize the provisional agreements reached at the informal plenary meeting on the draft annual report , as contained in document CD/WP.511 , together with the revisio
 ns contained in document CD/WP.512 . Since we were able to go through the draft annual report , paragraph by paragraph , at the informal plenary meeting held last Thursday , and since we have gone through the revisions contained in CD/WP.512 earlier this morning , I shall now proceed to the formal adoption of our annual report as a whole as revised . May I take it that the annual report in its entirety , as contained in document CD/WP.511 and as revised in CD/WP.512 , is adopted ? It is so decided . The secretariat will issue the report as an official document of the Conference in all official languages as soon as feasible . Does any delegation wish to take the floor at this stage ? I should like to inform the Conference that , as Cameroon has notified the secretariat that it will not be in a position to preside over the Conference from 1 January 2001 , as provided for in rule 9 of the rules of procedure , the representatives of the following member States will preside over the Conf
 erence in 2001 : Canada from 1 January to 18 February ; Chile from 19 February to 18 March ; China from 19 March to 27 May ; Colombia from 28 May to 24 June ; Cuba from 25 June to 19 August ; and the Democratic People 's Republic of Korea from 20 August to 31 December 2001. I would now like to give the floor to the Deputy Secretary-General of the Conference , Mr. Abdelkader Bensmail , who wishes to make a brief statement . Mr. BENSMAIL ( Deputy Secretary-General of the Conference on Disarmament ) ( translated from French ) : Mr. President , dear friends , as you know , I am about to leave my position as Deputy Secretary-General of the Conference on Disarmament and as Director of the Geneva Branch of the Department for Disarmament Affairs . I therefore wish to take advantage of this last plenary of the 2000 session to take my leave of you all and briefly to share a few personal reflections with you . Let me reassure you at the very outset : I do not intend to apply the tacit rule in 
 the Conference that the length of a valedictory statement should be one page per year of service with the Conference . I certainly do not intend to inflict a 20-page statement on you , nor do I intend to abide by the ritual but perilous exercise of drawing definitive conclusions on the future of the Conference . Others have done that before me and with greater authority , so I will merely make a few comments inspired by the long experience I have gathered in this forum . It is my great privilege to have been associated with the Conference on Disarmament since its very beginnings , following the first special session of the General Assembly on disarmament in June 1978. I was a member of the Algerian delegation which had the honour of being the first to preside over the Conference in January 1979 and thus helping to do the groundwork for its future work . My move from the Algerian diplomatic service to the secretariat here in March 1980 was thus a natural progression , because in fact
  I continued to deal with the same subjects , albeit from a different standpoint , since in that new office I was no longer defending the position of my Government but serving the interests of all delegations and working to seek consensus . The move was not an easy one but the counsel which I received from my superiors was of great assistance to me at the time . When one day I asked the first Secretary-General of the Conference , the late Ricki Jaipal , what the golden rule was for the secretariat , he said : \u201c The secretariat should be seen and not heard . \u201d I think I can say that I have scrupulously stuck to that rule throughout my service , except for one particular occasion when there was an attempt to question the secretariat 's integrity . Thus , I was present at the birth of this institution . I witnessed the immense hopes surrounding it , that it would help rid the world of the threat of weapons of mass destruction and help create an international community determined to
  reduce conventional weapons to levels compatible with the preservation of national security . I observed its hesitant steps as it tried to forge consensus on priorities for negotiation . I also lived through the Conference 's hours of glory , as it conducted the negotiations which led to the conclusion of the Chemical Weapons Convention in 1992 and the Comprehensive Nuclear Test-Ban Treaty in 1996 , to which processes I made my own modest contribution . And , finally , I have shared your frustration at the Conference 's inability to embark on substantive work on its agenda items . ( continued in English ) The ongoing debate on the role of the Conference in the field of multilateral disarmament diplomacy in the current international security environment is legitimate and necessary . In this connection , it should be recalled that periods of inactivity and deadlock have occurred before in the framework of the predecessors of the Conference on Disarmament , namely , the Ten-Nation Com
 mittee on Disarmament , the Eighteen-Nation Committee on Disarmament and the Conference of the Committee on Disarmament . I remember the days when questions were raised as to whether there was a life for the Conference on Disarmament after the conclusion of the Chemical Weapons Convention . These questions were , as I recall , answered with the decision of the Conference to negotiate the Comprehensive Nuclear Test-Ban Treaty . Similar questions are being asked now . Can the Conference overcome the impasse it currently faces , following the indefinite extension of the NPT and the conclusion of the CTBT ? Are we witnessing the emergence of a new model for arms control and disarmament with the \u201c Ottawa process \u201d , which led to the agreement on a convention to ban landmines ? Can this experience be repeated in other fields of arms control and disarmament ? Are diplomatic conferences convened through a coalition of like-minded Governments and non-governmental organizations , as a sub
 stitute for genuine multilateral negotiations , which are the only negotiations that can ensure universal adherence to treaties and their effective implementation ? The debate on all these questions is not over and is unlikely to be conclusive for the simple reason that the nature of the issues to be addressed in the field of arms control and disarmament , their complexity and magnitude , require that all possible avenues , whether bilateral , regional or global , are fully explored and used . The current deadlock in the Conference on Disarmament has been and continues to be the subject of various interpretations . While it is recognized that the Conference , as an institution , has served the international community well in the past , and that its potential for other multilateral negotiations remains intact , the prolongation of this stalemate is a source of concern to all . The Conference 's value cannot , and should not , be reduced to a simple accounting of the number of treatie
 s it produces . The situation it faces can be largely attributed to a fundamental divergence of views on the wider question of what is the international disarmament agenda after the end of the cold war , the conclusion of the Chemical Weapons Convention , the indefinite extension of the Non-Proliferation Treaty and the Comprehensive Nuclear Test-Ban Treaty . In particular , it relates to the requirement to strike the right balance between the preservation and consolidation of the nuclear non-proliferation regime and the need to accelerate and multilateralize the nuclear disarmament process , while at the same time preserving and enhancing strategic stability . Preparing the ground for future negotiations through discussions and technical work is a prerequisite for the start of genuine negotiations . All major negotiations have been preceded by a pre-negotiation stage , in which some shared understanding is reached that a security problem exists and that it must be addressed multilat
 erally . This process may be arduous and time-consuming , but it provides the guarantee that the end-product , that is to say , treaty-making , is based on solid foundations which take into account the security concerns of all and therefore ensures the universality and effectiveness of the agreements reached . What is required is a common willingness of all the membership of the Conference , making full use of its in-built flexibility and recognized expertise , to develop a workable and balanced programme of work which takes into account the priorities and concerns of all . Multilateral disarmament forums have always evolved over the years in response to changed political realities . The Conference on Disarmament is no exception in this respect , and the difficulties it now faces are not due to what is perceived by some as the rigidity of its rules of procedure , of its working methods and the group system . They are rather a reflection of the complexity and the dynamics of contempo
 rary international relations , and therefore all efforts should focus on the creation of a political climate conducive to the full use of the Conference as a negotiating forum , in particular on the restoration of a minimum harmony among the major players . ( continued in French ) I realize that I have strayed somewhat from my intention not to embark on serious political comment , but I am sure that you will not hold that against me . My long association with the Conference on Disarmament has been a valuable experience for me at all levels , personal , political , intellectual and cultural . The friendship and trust that you and your predecessors have always shown towards me has really made things very much easier for me and I am grateful to you for that . I would not want to conclude these comments without conveying my sincere thanks to the whole team which has given me such valuable assistance in carrying out my duties over all these years : Hannelore Hoppe , Silvana da Silva , Vl
 adimir Bogomolov , Jerzy Zaleski , Sonia Koppe , Alexandre Golay , Yvonne Santa Eugenia , Charlotte Laut Hern�ndez , Cheryl Darby , Annette Ekberg , Lynne Hardewall and Sa�d Zoughy were skilful , devoted colleagues , always ready to help and to put up with my Mediterranean character . And , finally , my thanks go to the interpreters , whom we so often put through the mill and who have always responded courteously to the sometimes unreasonable demands placed on them by the Conference . The PRESIDENT : I thank Mr. Bensmail for his profound and enlightening statement . Would any delegation like to take the floor at this point ? I see the representative of France . Ambassador de La Fortelle , you have the floor . Mr. de La FORTELLE ( France ) ( translated from French ) : Mr. President , my dear colleagues , the interpreters , ladies and gentlemen , the Deputy Secretary-General of the Conference on Disarmament , Abdelkader Bensmail , has just given us some sad news today : for him this
  will be the last plenary meeting of the Conference on Disarmament in Geneva . Fortunately , he will be accompanying many of us to New York to follow the work of the First Committee and will take his retirement thereafter . It is with great sadness that we will see him leave after a long career that has taken place , if I may say so , on both sides of the looking-glass : first , as a diplomat representing Algeria , and then as an international official in the United Nations . At the risk of offending his well-known modesty , I will recall the major stages of his university , professional and family life which have made of him both a great Algerian and a great Frenchman . Abdelkader Bensmail attained outstanding academic success in my country . He has a postgraduate degree in English and degrees from the Toulouse Institute of Political Studies and the Institute for Foreign Affairs in Paris . With such fine qualifications , a career in international affairs was unavoidable and it is i
 ndeed that career that he chose . Abdelkader Bensmail began his career in the Algerian diplomatic service and served his country with distinction . As he has recalled to us , he became an international civil servant with the United Nations first in New York , in the International Affairs Department , and then in Geneva in the Conference on Disarmament . In October 1993 , the Secretary-General of the United Nations appointed him to the post of Deputy Secretary-General of the Conference on Disarmament and head of the Geneva Branch of the United Nations Centre for Disarmament Affairs . As you can see , at the end of a career both varied and full , his outstanding merits have always received due appreciation and positions of high responsibility have been conferred upon him , in particular , the post which he has held for the last seven years here in Geneva . Abdelkader Bensmail has always acted in a manner both effective and discreet . He has rarely taken the floor , and he has told us 
 why , but his advice was always valuable , particularly for us . As he departs , he will be taking with him the memory of the Conference . We could always turn to him to learn of those ever useful precedents . My predecessors were able to appreciate his many qualities during both the negotiations of the Convention on Chemical Weapons and those of the CTBT , and I believe I speak for all of us when I say that we will all miss him in the time to come , as he enjoys his well-deserved retirement . It remains only for me to convey my very best wishes to the Deputy Secretary-General and to hope that he enjoys happiness and the best of health in his new life . The PRESIDENT : I thank the representative of France for his statement . Would any other delegation like to take the floor ? I see the representative of Egypt . Ambassador , you have the floor . Ms. ABOUNAGA ( Egypt ) ( translated from Arabic ) : Mr. President , since I come from the same region as Mr. Bensmail , might I be allowed t
 hrough you to say these few words to him ? On behalf of the Egyptian delegation , I would like to express our gratitude and appreciation to Mr. Abdelkader Bensmail , the Deputy Secretary-General of the Conference on Disarmament , who unfortunately is leaving us today to enjoy what I am sure will be a better life after his official duties and after all the dedicated services that he has rendered , for almost a quarter of a century , to his country , Algeria , and in furtherance of the aims of the Conference on Disarmament and the programmes and agenda for disarmament in general . As we and previous representatives of our countries to this Conference can testify , these services were rendered over the years in a highly competent manner . They were made possible by virtue of his commendable personal qualities , which he described , very modestly , as being possibly attributable to his Mediterranean character . They certainly turned him into a team leader who has worked with a great dea
 l of skill and efficiency . Today we bid him farewell , but only here in the Conference , and may I personally , on behalf of the Egyptian delegation , wish him every success in his future life . I am quite sure that , in mind , thought and spirit , he will not abandon the Conference on Disarmament or the objectives of disarmament and that he will always remain very close to us here in the Conference . I wish him all the best and reiterate my sincere gratitude and appreciation on behalf of the Egyptian delegation . Thank you . The PRESIDENT : I now give the floor to China . You have the floor , Ambassador Hu . Mr. HU XIAODI ( China ) ( translated from Chinese ) : Thank you , Mr. President . Like the two previous speakers , I shall also deeply regret the impending retirement from office of our Deputy Secretary-General , Mr. Bensmail . With his outstanding diplomatic skill and rich work experience , Mr. Bensmail has over the course of many years performed a great deal of valuable work
  for our Conference on Disarmament , and has rendered enormous assistance to all the delegations , for which he has earned the highest praise . In fact , since the very foundation of the Conference and throughout his work with this forum , he has served as a veritable living encyclopaedia of all the issues relating to the Conference on Disarmament . The statement which he has just delivered is full of wisdom and merits our deep consideration . Without question , his departure will be a loss both to the Conference on Disarmament and to us all . On behalf of the Chinese delegation , I would like to express our wholehearted appreciation for all the help that he has given us over all the years since we first joined the Conference on Disarmament . I also sincerely wish him every success for his future . The PRESIDENT : I now give the floor to the representative of Algeria . Mr. BENFREHA ( Algeria ) ( translated from French ) : As a fellow countryman , it is incumbent on me , on behalf of
  my country , to pay special tribute to Mr. Abdelkader Bensmail , Deputy Secretary-General of the Conference on Disarmament , for the quality of the work he has performed here in the Conference on Disarmament and , at the same time , to extol the efforts he has made since the establishment of this multilateral forum to instil in it a spirit of consensus . We see in Mr. Bensmail a consummate product of the Algerian diplomatic service , the service in which he embarked on his professional career . His exemplary approach to his work was forged in this school of diplomacy , which has always advocated the attainment of consensus in dealing with key issues of disarmament , in the interest of peace and international security . I would also like to associate myself with the eloquent praise expressed by those speaking before me for Mr. Bensmail 's outstanding qualities and to give him my best wishes for his retirement . The PRESIDENT : I thank the representative of Algeria for his comments .
  Does any other delegation wish to take the floor ? I see the representative of India . Mr. SOOD ( India ) : It is perhaps somewhat presumptuous for a person who has just joined the Conference on Disarmament to seek the floor twice on his very first day in this forum , but this is a rather special day , and I exercise this privilege as one of the people who have known Kader since 1986. It is both strange and somehow special that , as I attend my first plenary in this new appointment , it also happens to be Kader 's last plenary in his present appointment . Many people around this table have spoken about his professional skills , but what I would like first to single out in this enormously rewarding relationship , my personal relationship with Kader , are his generosity with his time , which he so willingly shared with me when I came here for the first time in 1986 , knowing very little about multilateral work and even less about disarmament , and the ease with which we could rely on
  him to give us the history and the background of issues and to educate us in the niceties of multilateral work . And second I recall his warm hospitality , which I have enjoyed on more than one occasion . The last memorable such occasion was the farewell dinner given in my own honour at his house in 1989. When I was appointed here , one of the things I looked forward to was the chance to resume this close contact , which we had kept up over the years , but within obvious limits , given that we were in different cities . All the same , I am glad that I have been present today at your last plenary meeting in this appointment . I would like to wish you the best of luck and good health , happiness and contentment . The PRESIDENT : I thank the representative of India for his statement . I now give the floor to the representative of the Republic of Korea . Mr. SUL ( Republic of Korea ) : Since this is the first time that I am taking the floor under your presidency , I should like to cong
 ratulate you on your assumption of the presidency . Please be assured of my delegation 's full support in carrying out your noble endeavours . I would also like to express my appreciation to Mr. Petrovsky , Secretary-General of the Conference , and to the staff of the secretariat for their excellent job in providing conference services . I have asked for the floor today on behalf of the members of the Western Group to express our regret and deep sadness over Mr. Bensmail 's departure . It is very difficult for me to add anything to the eloquent tributes already paid to Mr. Bensmail this morning . I would like to say , however , how much we have benefited from his vast knowledge and experience in disarmament negotiations , his diplomatic skills and the managerial proficiency which he brought to the Conference . Mr. Bensmail assumed many roles , sometimes as an untiring negotiator , sometimes as a diligent provider of a good and constructive atmosphere for the work of the Conference .
  His charming personality and engaging character have constantly stimulated the work of the Conference . He was a good friend of the Western Group and , I am sure , of all the delegations to the Conference on Disarmament . We all agree that , with his years of experience and dedication , Mr. Bensmail has become a \u201c walking library \u201d of the Conference , given his extensive knowledge not only of all those complicated procedural issues but also of many aspects never recorded in any document . How reassuring it was to know that we could always count on Mr. Bensmail effortlessly to solve what seemed to us unfathomable mysteries . The Western Group expresses its most sincere gratitude to Mr. Bensmail for his dedication and outstanding professionalism . We will miss him very much , but I find consolation in a Korean proverb which wisely says that , while those who meet are destined to part , those who part are destined to meet again . We wish him all the best , continued good health an
 d happiness . The PRESIDENT : I thank the representative of the Republic of Korea for his statement and for the kind words addressed to the Chair . I now give the floor to the representative of Romania . Mr. HORUMBA ( Romania ) : Mr. President , speaking on behalf of the Eastern European Group , I would like wholeheartedly to endorse the kind words of appreciation addressed by previous speakers to Mr. Abdelkader Bensmail . The Eastern European Group would also like to thank Mr. Bensmail for his efforts in the important office of Deputy Secretary-General of the Conference on Disarmament and to commend him on his long-standing experience and expertise in the field of multilateral disarmament , his diplomatic skills and the managerial competence he has displayed in the activities of our unique international body . As we bid farewell to Mr. Abdelkader Bensmail , we wish him every success in the years ahead , long health and prosperity , as well as personal happiness . The PRESIDENT : As
  I see no other speakers , I shall proceed with some short closing remarks . In my opening remarks I identified two principal objectives in my work as the last presiding officer of the 2000 session of the Conference on Disarmament : to continue the efforts towards finding a compromise on the programme or work and to facilitate the adoption of the Conference 's report to the General Assembly of the United Nations . The second task has just been accomplished , and I am grateful to all delegations for the assistance and the collaboration that have made this possible . I believe that the report which we have adopted is factual and accurate in accounting for the Conference 's activities in the year 2000. With regard to the first task , I am compelled to conclude today , on behalf of this Conference , that we have spent yet another session mostly on trying to pre-negotiate the conditions for our possible future negotiations on the different topics of our agenda , upon which we happen to a
 gree . My consultations to date have confirmed that we are still short of a compromise on a programme of work for the Conference . While there is a broad measure of agreement on most of the elements for such a programme , there are two issues , nuclear disarmament and the prevention of an arms race in outer space , that have still to be tackled further . In addition , to be able to agree about anything in the programe of work , it seems that first we have to agree about everything . There were times when the situation seemed so serious that , as the saying goes , we could only joke about it . It is clear that we have before us a tall order and there should be nothing exceptional about the fact that it is taking us so long to perform . That having been said , these past four weeks I have also become aware of a number of promising signs in the Conference on Disarmament . Of the two outstanding issues on our draft programme of work , our positions on nuclear disarmament seem to be a li
 ttle closer to convergence today than they were about a year ago . In addition , most of the delegations with which I have had meetings share the same concern and similar perceptions with regard to the delicate stage at which we find ourselves in the Conference on Disarmament . The positive attitude of all members in preparing our annual report to the General Assembly , as well as the contents of that report , are also testimony to the good will prevailing in the Conference for our endeavour to surmount the hurdles that face us . Our joint efforts brought about an encouraging development in our report CD/1624 , as the accumulation of new constructive ideas expressed in the Conference during the current and previous years . The report that we adopted delivers a forward-looking recommendation to me and to my successor to conduct intensive consultations during the inter-sessional period on the basis of the progress that has already been achieved . I intend those consultations to be ope
 n , pragmatic and cooperative , with the sole objective of preparing a good beginning for the 2001 session . All in all , provided we are able to demonstrate the necessary political will , we may well be just a couple of critical steps away from the desired compromise and the start of substantive work in the Conference on Disarmament . As in a long-distance race , the last few steps often prove to be the most difficult . But I believe that these steps will not be impossible to make ; we are , after all , carrying the baton forward into the next millennium ( please forgive the clich� , but desperate times call for desperate metaphors ) . At this threshold in time , we the peoples are striving for a safer world , based on development and disarmament . We the peoples prefer to compete in sporting competition , rather than arms races . Our heads of State and Government exercised their political will in resolving , earlier this month , to share the responsibility for international peace
  and security , assigning special significance to disarmament , amongst other areas . The Conference on Disarmament , as the sole multilateral negotiating forum in its field , has a unique role to play in delivering on this commitment . In conclusion , I would like to express my gratitude to the Secretary-General of the Conference , Mr. Vladimir Petrovsky , who today had to attend a memorial service and unfortunately could not be with us ; to all the members of the secretariat for their efficiency and valuable advice ; to the interpreters , for their patience and dedication ; and to all delegations for their cooperation and understanding . Today is also a special occasion for us . It is my honour and my privilege to pay tribute , on behalf of all the members of this Conference , to the Deputy Secretary-General , Mr. Abdelkader Bensmail , who will be retiring later this year . Mr. Bensmail has had an illustrious diplomatic career , the last 20 and more years of which have been devote
 d to the Conference on Disarmament and to disarmament in general . His expert guidance and professional competence have been emblematic of the work of the secretariat and will remain as an example to follow . He is what I would call the \u201c living history \u201d of this Conference , and I can only hope that some day he will put it all down on paper . Mr. Bensmail , your experience and outstanding contribution will be greatly missed by us all . We wish you and your family good health , good luck and continued success . The next plenary meeting of the Conference will be held on Tuesday , 23 January 2001 , at 10 a. m. The meeting rose at 12.15 p. m. 
\ No newline at end of file


[41/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib1/27Bib1Amos_Prophecy_12_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib1/27Bib1Amos_Prophecy_12_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib1/27Bib1Amos_Prophecy_12_EN.txt.txt
new file mode 100644
index 0000000..0425235
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib1/27Bib1Amos_Prophecy_12_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Amos 1 1 The words of Amos , one of the shepherds of Tekoa\u2014what he saw concerning Israel two years before the earthquake , when Uzziah was king of Judah and Jeroboam son of Jehoash[a ] was king of Israel . 2 He said : \u201c The LORD roars from Zion and thunders from Jerusalem ; the pastures of the shepherds dry up , [ b ] and the top of Carmel withers . \u201d Judgment on Israel\u2019s Neighbors 3 This is what the LORD says : \u201c For three sins of Damascus , even for four , I will not turn back my wrath . Because she threshed Gilead with sledges having iron teeth , 4 I will send fire upon the house of Hazael that will consume the fortresses of Ben-Hadad . 5 I will break down the gate of Damascus ; I will destroy the king who is in[c ] the Valley of Aven[d ] and the one who holds the scepter in Beth Eden . The people of Aram will go into exile to Kir , \u201d says the LORD . 6 This is what the LORD says : \u201c For three sins of Gaza , even for four , I will not turn back my wrath . Because
  she took captive whole communities and sold them to Edom , 7 I will send fire upon the walls of Gaza that will consume her fortresses . 8 I will destroy the king[e ] of Ashdod and the one who holds the scepter in Ashkelon . I will turn my hand against Ekron , till the last of the Philistines is dead , \u201d says the Sovereign LORD . 9 This is what the LORD says : \u201c For three sins of Tyre , even for four , I will not turn back my wrath . Because she sold whole communities of captives to Edom , disregarding a treaty of brotherhood , 10 I will send fire upon the walls of Tyre that will consume her fortresses . \u201d 11 This is what the LORD says : \u201c For three sins of Edom , even for four , I will not turn back my wrath . Because he pursued his brother with a sword , stifling all compassion , [ f ] because his anger raged continually and his fury flamed unchecked , 12 I will send fire upon Teman that will consume the fortresses of Bozrah . \u201d 13 This is what the LORD says : \u201c For th
 ree sins of Ammon , even for four , I will not turn back my wrath . Because he ripped open the pregnant women of Gilead in order to extend his borders , 14 I will set fire to the walls of Rabbah that will consume her fortresses amid war cries on the day of battle , amid violent winds on a stormy day . 15 Her king[g ] will go into exile , he and his officials together , \u201d says the LORD . ** Amos 2 1 This is what the LORD says : \u201c For three sins of Moab , even for four , I will not turn back my wrath . Because he burned , as if to lime , the bones of Edom\u2019s king , 2 I will send fire upon Moab that will consume the fortresses of Kerioth . [ a ] Moab will go down in great tumult amid war cries and the blast of the trumpet . 3 I will destroy her ruler and kill all her officials with him , \u201d says the LORD . 4 This is what the LORD says : \u201c For three sins of Judah , even for four , I will not turn back my wrath . Because they have rejected the law of the LORD and have not kept hi
 s decrees , because they have been led astray by false gods , [ b ] the gods[c ] their ancestors followed , 5 I will send fire upon Judah that will consume the fortresses of Jerusalem . \u201d Judgment on Israel 6 This is what the LORD says : \u201c For three sins of Israel , even for four , I will not turn back my wrath . They sell the righteous for silver , and the needy for a pair of sandals . 7 They trample on the heads of the poor as upon the dust of the ground and deny justice to the oppressed . Father and son use the same girl and so profane my holy name . 8 They lie down beside every altar on garments taken in pledge . In the house of their god they drink wine taken as fines . 9 \u201c I destroyed the Amorite before them , though he was tall as the cedars and strong as the oaks . I destroyed his fruit above and his roots below . 10 \u201c I brought you up out of Egypt , and I led you forty years in the desert to give you the land of the Amorites . 11 I also raised up prophets from among
  your sons and Nazirites from among your young men . Is this not true , people of Israel ? \u201d declares the LORD . 12 \u201c But you made the Nazirites drink wine and commanded the prophets not to prophesy . 13 \u201c Now then , I will crush you as a cart crushes when loaded with grain . 14 The swift will not escape , the strong will not muster their strength , and the warrior will not save his life . 15 The archer will not stand his ground , the fleet-footed soldier will not get away , and the horseman will not save his life . 16 Even the bravest warriors will flee naked on that day , \u201d declares the LORD . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib1/28Bib1Amos_Prophecy_89_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib1/28Bib1Amos_Prophecy_89_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib1/28Bib1Amos_Prophecy_89_EN.txt.txt
new file mode 100644
index 0000000..29a8b31
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib1/28Bib1Amos_Prophecy_89_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Amos 8 A Basket of Ripe Fruit 1 This is what the Sovereign LORD showed me : a basket of ripe fruit . 2 \u201c What do you see , Amos ? \u201d he asked . \u201c A basket of ripe fruit , \u201d I answered . Then the LORD said to me , \u201c The time is ripe for my people Israel ; I will spare them no longer . 3 \u201c In that day , \u201d declares the Sovereign LORD , \u201c the songs in the temple will turn to wailing . [ a ] Many , many bodies\u2014flung everywhere ! Silence ! \u201d 4 Hear this , you who trample the needy and do away with the poor of the land , 5 saying , \u201c When will the New Moon be over that we may sell grain , and the Sabbath be ended that we may market wheat ? \u201d \u2014 skimping the measure , boosting the price and cheating with dishonest scales , 6 buying the poor with silver and the needy for a pair of sandals , selling even the sweepings with the wheat . 7 The LORD has sworn by the Pride of Jacob : \u201c I will never forget anything they have done . 8 \u201c Will not the land tremble for t
 his , and all who live in it mourn ? The whole land will rise like the Nile ; it will be stirred up and then sink like the river of Egypt . 9 \u201c In that day , \u201d declares the Sovereign LORD , \u201c I will make the sun go down at noon and darken the earth in broad daylight . 10 I will turn your religious feasts into mourning and all your singing into weeping . I will make all of you wear sackcloth and shave your heads . I will make that time like mourning for an only son and the end of it like a bitter day . 11 \u201c The days are coming , \u201d declares the Sovereign LORD , \u201c when I will send a famine through the land \u2014 not a famine of food or a thirst for water , but a famine of hearing the words of the LORD . 12 Men will stagger from sea to sea and wander from north to east , searching for the word of the LORD , but they will not find it . 13 \u201c In that day \u201c the lovely young women and strong young men will faint because of thirst . 14 They who swear by the shame[b ] of Samaria 
 , or say , \u2018 As surely as your god lives , O Dan , ' or , \u2018 As surely as the god[c ] of Beersheba lives ' \u2014 they will fall , never to rise again . \u201d Amos 9 Israel to Be Destroyed 1 I saw the Lord standing by the altar , and he said : \u201c Strike the tops of the pillars so that the thresholds shake . Bring them down on the heads of all the people ; those who are left I will kill with the sword . Not one will get away , none will escape . 2 Though they dig down to the depths of the grave , [ a ] from there my hand will take them . Though they climb up to the heavens , from there I will bring them down . 3 Though they hide themselves on the top of Carmel , there I will hunt them down and seize them . Though they hide from me at the bottom of the sea , there I will command the serpent to bite them . 4 Though they are driven into exile by their enemies , there I will command the sword to slay them . I will fix my eyes upon them for evil and not for good . \u201d 5 The Lord , the LORD
  Almighty , he who touches the earth and it melts , and all who live in it mourn \u2014 the whole land rises like the Nile , then sinks like the river of Egypt \u2014 6 he who builds his lofty palace[b ] in the heavens and sets its foundation[c ] on the earth , who calls for the waters of the sea and pours them out over the face of the land \u2014 the LORD is his name . 7 \u201c Are not you Israelites the same to me as the Cushites[d ] ? \u201d declares the LORD . \u201c Did I not bring Israel up from Egypt , the Philistines from Caphtor[e ] and the Arameans from Kir ? 8 \u201c Surely the eyes of the Sovereign LORD are on the sinful kingdom . I will destroy it from the face of the earth \u2014 yet I will not totally destroy the house of Jacob , \u201d declares the LORD . 9 \u201c For I will give the command , and I will shake the house of Israel among all the nations as grain is shaken in a sieve , and not a pebble will reach the ground . 10 All the sinners among my people will die by the sword , all those who s
 ay , \u2018 Disaster will not overtake or meet us . ' Israel 's Restoration 11 \u201c In that day I will restore David 's fallen tent . I will repair its broken places , restore its ruins , and build it as it used to be , 12 so that they may possess the remnant of Edom and all the nations that bear my name , [ f ] \u201d declares the LORD , who will do these things . 13 \u201c The days are coming , \u201d declares the LORD , \u201c when the reaper will be overtaken by the plowman and the planter by the one treading grapes . New wine will drip from the mountains and flow from all the hills . 14 I will bring back my exiled[g ] people Israel ; they will rebuild the ruined cities and live in them . They will plant vineyards and drink their wine ; they will make gardens and eat their fruit . 15 I will plant Israel in their own land , never again to be uprooted from the land I have given them , \u201d says the LORD your God . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib1/29Bib1Ezra_History_12_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib1/29Bib1Ezra_History_12_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib1/29Bib1Ezra_History_12_EN.txt.txt
new file mode 100644
index 0000000..5a2997b
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib1/29Bib1Ezra_History_12_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Ezra 1 Cyrus Helps the Exiles to Return 1 In the first year of Cyrus king of Persia , in order to fulfill the word of the LORD spoken by Jeremiah , the LORD moved the heart of Cyrus king of Persia to make a proclamation throughout his realm and to put it in writing : 2 \u201c This is what Cyrus king of Persia says : \u201c \u2018 The LORD , the God of heaven , has given me all the kingdoms of the earth and he has appointed me to build a temple for him at Jerusalem in Judah . 3 Anyone of his people among you\u2014may his God be with him , and let him go up to Jerusalem in Judah and build the temple of the LORD , the God of Israel , the God who is in Jerusalem . 4 And the people of any place where survivors may now be living are to provide him with silver and gold , with goods and livestock , and with freewill offerings for the temple of God in Jerusalem . \u2019 \u201d 5 Then the family heads of Judah and Benjamin , and the priests and Levites\u2014everyone whose heart God had moved\u2014prepared to go 
 up and build the house of the LORD in Jerusalem . 6 All their neighbors assisted them with articles of silver and gold , with goods and livestock , and with valuable gifts , in addition to all the freewill offerings . 7 Moreover , King Cyrus brought out the articles belonging to the temple of the LORD , which Nebuchadnezzar had carried away from Jerusalem and had placed in the temple of his god . [ a ] 8 Cyrus king of Persia had them brought by Mithredath the treasurer , who counted them out to Sheshbazzar the prince of Judah . 9 This was the inventory : gold dishes 30 silver dishes 1,000 silver pans[b ] 29 10 gold bowls 30 matching silver bowls 410 other articles 1,000 11 In all , there were 5,400 articles of gold and of silver . Sheshbazzar brought all these along when the exiles came up from Babylon to Jerusalem . Ezra 2 The List of the Exiles Who Returned 1 Now these are the people of the province who came up from the captivity of the exiles , whom Nebuchadnezzar king of Babylon
  had taken captive to Babylon ( they returned to Jerusalem and Judah , each to his own town , 2 in company with Zerubbabel , Jeshua , Nehemiah , Seraiah , Reelaiah , Mordecai , Bilshan , Mispar , Bigvai , Rehum and Baanah ) : The list of the men of the people of Israel : 3 the descendants of Parosh 2,172 4 of Shephatiah 372 5 of Arah 775 6 of Pahath-Moab ( through the line of Jeshua and Joab ) 2,812 7 of Elam 1,254 8 of Zattu 945 9 of Zaccai 760 10 of Bani 642 11 of Bebai 623 12 of Azgad 1,222 13 of Adonikam 666 14 of Bigvai 2,056 15 of Adin 454 16 of Ater ( through Hezekiah ) 98 17 of Bezai 323 18 of Jorah 112 19 of Hashum 223 20 of Gibbar 95 21 the men of Bethlehem 123 22 of Netophah 56 23 of Anathoth 128 24 of Azmaveth 42 25 of Kiriath Jearim , [ a ] Kephirah and Beeroth 743 26 of Ramah and Geba 621 27 of Micmash 122 28 of Bethel and Ai 223 29 of Nebo 52 30 of Magbish 156 31 of the other Elam 1,254 32 of Harim 320 33 of Lod , Hadid and Ono 725 34 of Jericho 345 35 of Senaah 3,630
  36 The priests : the descendants of Jedaiah ( through the family of Jeshua ) 973 37 of Immer 1,052 38 of Pashhur 1,247 39 of Harim 1,017 40 The Levites : the descendants of Jeshua and Kadmiel ( through the line of Hodaviah ) 74 41 The singers : the descendants of Asaph 128 42 The gatekeepers of the temple : the descendants of Shallum , Ater , Talmon , Akkub , Hatita and Shobai 139 43 The temple servants : the descendants of Ziha , Hasupha , Tabbaoth , 44 Keros , Siaha , Padon , 45 Lebanah , Hagabah , Akkub , 46 Hagab , Shalmai , Hanan , 47 Giddel , Gahar , Reaiah , 48 Rezin , Nekoda , Gazzam , 49 Uzza , Paseah , Besai , 50 Asnah , Meunim , Nephusim , 51 Bakbuk , Hakupha , Harhur , 52 Bazluth , Mehida , Harsha , 53 Barkos , Sisera , Temah , 54 Neziah and Hatipha 55 The descendants of the servants of Solomon : the descendants of Sotai , Hassophereth , Peruda , 56 Jaala , Darkon , Giddel , 57 Shephatiah , Hattil , Pokereth-Hazzebaim and Ami 58 The temple servants and the descendants o
 f the servants of Solomon 392 59 The following came up from the towns of Tel Melah , Tel Harsha , Kerub , Addon and Immer , but they could not show that their families were descended from Israel : 60 The descendants of Delaiah , Tobiah and Nekoda 652 61 And from among the priests : The descendants of Hobaiah , Hakkoz and Barzillai ( a man who had married a daughter of Barzillai the Gileadite and was called by that name ) . 62 These searched for their family records , but they could not find them and so were excluded from the priesthood as unclean . 63 The governor ordered them not to eat any of the most sacred food until there was a priest ministering with the Urim and Thummim . 64 The whole company numbered 42,360 , 65 besides their 7,337 menservants and maidservants ; and they also had 200 men and women singers . 66 They had 736 horses , 245 mules , 67 435 camels and 6,720 donkeys . 68 When they arrived at the house of the LORD in Jerusalem , some of the heads of the families gave
  freewill offerings toward the rebuilding of the house of God on its site . 69 According to their ability they gave to the treasury for this work 61,000 drachmas[b ] of gold , 5,000 minas[c ] of silver and 100 priestly garments . 70 The priests , the Levites , the singers , the gatekeepers and the temple servants settled in their own towns , along with some of the other people , and the rest of the Israelites settled in their towns . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib1/30Bib1Joel_Prophecy_12_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib1/30Bib1Joel_Prophecy_12_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib1/30Bib1Joel_Prophecy_12_EN.txt.txt
new file mode 100644
index 0000000..3430b8c
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib1/30Bib1Joel_Prophecy_12_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Joel 1 1 The word of the LORD that came to Joel son of Pethuel . An Invasion of Locusts 2 Hear this , you elders ; listen , all who live in the land . Has anything like this ever happened in your days or in the days of your forefathers ? 3 Tell it to your children , and let your children tell it to their children , and their children to the next generation . 4 What the locust swarm has left the great locusts have eaten ; what the great locusts have left the young locusts have eaten ; what the young locusts have left other locusts[a ] have eaten . 5 Wake up , you drunkards , and weep ! Wail , all you drinkers of wine ; wail because of the new wine , for it has been snatched from your lips . 6 A nation has invaded my land , powerful and without number ; it has the teeth of a lion , the fangs of a lioness . 7 It has laid waste my vines and ruined my fig trees . It has stripped off their bark and thrown it away , leaving their branches white . 8 Mourn like a virgin[b ] in sackcloth 
 grieving for the husband[c ] of her youth . 9 Grain offerings and drink offerings are cut off from the house of the LORD . The priests are in mourning , those who minister before the LORD . 10 The fields are ruined , the ground is dried up[d ] ; the grain is destroyed , the new wine is dried up , the oil fails . 11 Despair , you farmers , wail , you vine growers ; grieve for the wheat and the barley , because the harvest of the field is destroyed . 12 The vine is dried up and the fig tree is withered ; the pomegranate , the palm and the apple tree \u2014 all the trees of the field\u2014are dried up . Surely the joy of mankind is withered away . A Call to Repentance 13 Put on sackcloth , O priests , and mourn ; wail , you who minister before the altar . Come , spend the night in sackcloth , you who minister before my God ; for the grain offerings and drink offerings are withheld from the house of your God . 14 Declare a holy fast ; call a sacred assembly . Summon the elders and all who liv
 e in the land to the house of the LORD your God , and cry out to the LORD . 15 Alas for that day ! For the day of the LORD is near ; it will come like destruction from the Almighty . [ e ] 16 Has not the food been cut off before our very eyes \u2014 joy and gladness from the house of our God ? 17 The seeds are shriveled beneath the clods . [ f ] The storehouses are in ruins , the granaries have been broken down , for the grain has dried up . 18 How the cattle moan ! The herds mill about because they have no pasture ; even the flocks of sheep are suffering . 19 To you , O LORD , I call , for fire has devoured the open pastures and flames have burned up all the trees of the field . 20 Even the wild animals pant for you ; the streams of water have dried up and fire has devoured the open pastures . Joel 2 An Army of Locusts 1 Blow the trumpet in Zion ; sound the alarm on my holy hill . Let all who live in the land tremble , for the day of the LORD is coming . It is close at hand \u2014 2 a da
 y of darkness and gloom , a day of clouds and blackness . Like dawn spreading across the mountains a large and mighty army comes , such as never was of old nor ever will be in ages to come . 3 Before them fire devours , behind them a flame blazes . Before them the land is like the garden of Eden , behind them , a desert waste \u2014 nothing escapes them . 4 They have the appearance of horses ; they gallop along like cavalry . 5 With a noise like that of chariots they leap over the mountaintops , like a crackling fire consuming stubble , like a mighty army drawn up for battle . 6 At the sight of them , nations are in anguish ; every face turns pale . 7 They charge like warriors ; they scale walls like soldiers . They all march in line , not swerving from their course . 8 They do not jostle each other ; each marches straight ahead . They plunge through defenses without breaking ranks . 9 They rush upon the city ; they run along the wall . They climb into the houses ; like thieves they en
 ter through the windows . 10 Before them the earth shakes , the sky trembles , the sun and moon are darkened , and the stars no longer shine . 11 The LORD thunders at the head of his army ; his forces are beyond number , and mighty are those who obey his command . The day of the LORD is great ; it is dreadful . Who can endure it ? Rend Your Heart 12 \u201c Even now , \u201d declares the LORD , \u201c return to me with all your heart , with fasting and weeping and mourning . \u201d 13 Rend your heart and not your garments . Return to the LORD your God , for he is gracious and compassionate , slow to anger and abounding in love , and he relents from sending calamity . 14 Who knows ? He may turn and have pity and leave behind a blessing \u2014 grain offerings and drink offerings for the LORD your God . 15 Blow the trumpet in Zion , declare a holy fast , call a sacred assembly . 16 Gather the people , consecrate the assembly ; bring together the elders , gather the children , those nursing at the brea
 st . Let the bridegroom leave his room and the bride her chamber . 17 Let the priests , who minister before the LORD , weep between the temple porch and the altar . Let them say , \u201c Spare your people , O LORD . Do not make your inheritance an object of scorn , a byword among the nations . Why should they say among the peoples , \u2018 Where is their God ? ' \u201d The LORD 's Answer 18 Then the LORD will be jealous for his land and take pity on his people . 19 The LORD will reply[a ] to them : \u201c I am sending you grain , new wine and oil , enough to satisfy you fully ; never again will I make you an object of scorn to the nations . 20 \u201c I will drive the northern army far from you , pushing it into a parched and barren land , with its front columns going into the eastern sea[b ] and those in the rear into the western sea . [ c ] And its stench will go up ; its smell will rise . \u201d Surely he has done great things . [ d ] 21 Be not afraid , O land ; be glad and rejoice . Surely the LOR
 D has done great things . 22 Be not afraid , O wild animals , for the open pastures are becoming green . The trees are bearing their fruit ; the fig tree and the vine yield their riches . 23 Be glad , O people of Zion , rejoice in the LORD your God , for he has given you the autumn rains in righteousness . [ e ] He sends you abundant showers , both autumn and spring rains , as before . 24 The threshing floors will be filled with grain ; the vats will overflow with new wine and oil . 25 \u201c I will repay you for the years the locusts have eaten \u2014 the great locust and the young locust , the other locusts and the locust swarm[f ] \u2014 my great army that I sent among you . 26 You will have plenty to eat , until you are full , and you will praise the name of the LORD your God , who has worked wonders for you ; never again will my people be shamed . 27 Then you will know that I am in Israel , that I am the LORD your God , and that there is no other ; never again will my people be shamed .
  The Day of the LORD 28 \u201c And afterward , I will pour out my Spirit on all people . Your sons and daughters will prophesy , your old men will dream dreams , your young men will see visions . 29 Even on my servants , both men and women , I will pour out my Spirit in those days. 30 I will show wonders in the heavens and on the earth , blood and fire and billows of smoke . 31 The sun will be turned to darkness and the moon to blood before the coming of the great and dreadful day of the LORD . 32 And everyone who calls on the name of the LORD will be saved ; for on Mount Zion and in Jerusalem there will be deliverance , as the LORD has said , among the survivors whom the LORD calls . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib1/31Bib1Ruth_History_12_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib1/31Bib1Ruth_History_12_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib1/31Bib1Ruth_History_12_EN.txt.txt
new file mode 100644
index 0000000..b4e6845
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib1/31Bib1Ruth_History_12_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Ruth 1 Naomi and Ruth 1 In the days when the judges ruled , [ a ] there was a famine in the land , and a man from Bethlehem in Judah , together with his wife and two sons , went to live for a while in the country of Moab . 2 The man 's name was Elimelech , his wife 's name Naomi , and the names of his two sons were Mahlon and Kilion . They were Ephrathites from Bethlehem , Judah . And they went to Moab and lived there . 3 Now Elimelech , Naomi 's husband , died , and she was left with her two sons . 4 They married Moabite women , one named Orpah and the other Ruth . After they had lived there about ten years , 5 both Mahlon and Kilion also died , and Naomi was left without her two sons and her husband . 6 When she heard in Moab that the LORD had come to the aid of his people by providing food for them , Naomi and her daughters-in-law prepared to return home from there . 7 With her two daughters-in-law she left the place where she had been living and set out on the road that woul
 d take them back to the land of Judah . 8 Then Naomi said to her two daughters-in-law , \u201c Go back , each of you , to your mother 's home . May the LORD show kindness to you , as you have shown to your dead and to me . 9 May the LORD grant that each of you will find rest in the home of another husband . \u201d Then she kissed them and they wept aloud 10 and said to her , \u201c We will go back with you to your people . \u201d 11 But Naomi said , \u201c Return home , my daughters . Why would you come with me ? Am I going to have any more sons , who could become your husbands ? 12 Return home , my daughters ; I am too old to have another husband . Even if I thought there was still hope for me\u2014even if I had a husband tonight and then gave birth to sons \u2014 13 would you wait until they grew up ? Would you remain unmarried for them ? No , my daughters . It is more bitter for me than for you , because the LORD 's hand has gone out against me ! \u201d 14 At this they wept again . Then Orpah kissed her
  mother-in-law good-by , but Ruth clung to her . 15 \u201c Look , \u201d said Naomi , \u201c your sister-in-law is going back to her people and her gods . Go back with her . \u201d 16 But Ruth replied , \u201c Do n't urge me to leave you or to turn back from you . Where you go I will go , and where you stay I will stay . Your people will be my people and your God my God . 17 Where you die I will die , and there I will be buried . May the LORD deal with me , be it ever so severely , if anything but death separates you and me . \u201d 18 When Naomi realized that Ruth was determined to go with her , she stopped urging her . 19 So the two women went on until they came to Bethlehem . When they arrived in Bethlehem , the whole town was stirred because of them , and the women exclaimed , \u201c Can this be Naomi ? \u201d 20 \u201c Do n't call me Naomi , [ b ] \u201d she told them . \u201c Call me Mara , [ c ] because the Almighty[d ] has made my life very bitter . 21 I went away full , but the LORD has brought me back emp
 ty . Why call me Naomi ? The LORD has afflicted[e ] me ; the Almighty has brought misfortune upon me . \u201d 22 So Naomi returned from Moab accompanied by Ruth the Moabitess , her daughter-in-law , arriving in Bethlehem as the barley harvest was beginning . Ruth 2 Ruth Meets Boaz 1 Now Naomi had a relative on her husband 's side , from the clan of Elimelech , a man of standing , whose name was Boaz . 2 And Ruth the Moabitess said to Naomi , \u201c Let me go to the fields and pick up the leftover grain behind anyone in whose eyes I find favor . \u201d Naomi said to her , \u201c Go ahead , my daughter . \u201d 3 So she went out and began to glean in the fields behind the harvesters . As it turned out , she found herself working in a field belonging to Boaz , who was from the clan of Elimelech . 4 Just then Boaz arrived from Bethlehem and greeted the harvesters , \u201c The LORD be with you ! \u201d \u201c The LORD bless you ! \u201d they called back . 5 Boaz asked the foreman of his harvesters , \u201c Whose youn
 g woman is that ? \u201d 6 The foreman replied , \u201c She is the Moabitess who came back from Moab with Naomi . 7 She said , \u2018 Please let me glean and gather among the sheaves behind the harvesters . ' She went into the field and has worked steadily from morning till now , except for a short rest in the shelter . \u201d 8 So Boaz said to Ruth , \u201c My daughter , listen to me . Do n't go and glean in another field and do n't go away from here . Stay here with my servant girls . 9 Watch the field where the men are harvesting , and follow along after the girls . I have told the men not to touch you . And whenever you are thirsty , go and get a drink from the water jars the men have filled . \u201d 10 At this , she bowed down with her face to the ground . She exclaimed , \u201c Why have I found such favor in your eyes that you notice me\u2014a foreigner ? \u201d 11 Boaz replied , \u201c I 've been told all about what you have done for your mother-in-law since the death of your husband\u2014how you left your f
 ather and mother and your homeland and came to live with a people you did not know before . 12 May the LORD repay you for what you have done . May you be richly rewarded by the LORD , the God of Israel , under whose wings you have come to take refuge . \u201d 13 \u201c May I continue to find favor in your eyes , my lord , \u201d she said . \u201c You have given me comfort and have spoken kindly to your servant\u2014though I do not have the standing of one of your servant girls . \u201d 14 At mealtime Boaz said to her , \u201c Come over here . Have some bread and dip it in the wine vinegar . \u201d When she sat down with the harvesters , he offered her some roasted grain . She ate all she wanted and had some left over . 15 As she got up to glean , Boaz gave orders to his men , \u201c Even if she gathers among the sheaves , do n't embarrass her . 16 Rather , pull out some stalks for her from the bundles and leave them for her to pick up , and do n't rebuke her . \u201d 17 So Ruth gleaned in the field until evening
  . Then she threshed the barley she had gathered , and it amounted to about an ephah . [ a ] 18 She carried it back to town , and her mother-in-law saw how much she had gathered . Ruth also brought out and gave her what she had left over after she had eaten enough . 19 Her mother-in-law asked her , \u201c Where did you glean today ? Where did you work ? Blessed be the man who took notice of you ! \u201d Then Ruth told her mother-in-law about the one at whose place she had been working . \u201c The name of the man I worked with today is Boaz , \u201d she said . 20 \u201c The LORD bless him ! \u201d Naomi said to her daughter-in-law . \u201c He has not stopped showing his kindness to the living and the dead . \u201d She added , \u201c That man is our close relative ; he is one of our kinsman-redeemers . \u201d 21 Then Ruth the Moabitess said , \u201c He even said to me , \u2018 Stay with my workers until they finish harvesting all my grain . ' \u201d 22 Naomi said to Ruth her daughter-in-law , \u201c It will be good for you ,
  my daughter , to go with his girls , because in someone else 's field you might be harmed . \u201d 23 So Ruth stayed close to the servant girls of Boaz to glean until the barley and wheat harvests were finished . And she lived with her mother-in-law . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib2/32Bib2John_Gospel_12_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib2/32Bib2John_Gospel_12_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib2/32Bib2John_Gospel_12_EN.txt.txt
new file mode 100644
index 0000000..516683a
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib2/32Bib2John_Gospel_12_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff John 1 The Word Became Flesh 1 In the beginning was the Word , and the Word was with God , and the Word was God . 2 He was with God in the beginning . 3 Through him all things were made ; without him nothing was made that has been made . 4 In him was life , and that life was the light of men . 5 The light shines in the darkness , but the darkness has not understood[a ] it . 6 There came a man who was sent from God ; his name was John . 7 He came as a witness to testify concerning that light , so that through him all men might believe . 8 He himself was not the light ; he came only as a witness to the light . 9 The true light that gives light to every man was coming into the world . [ b ] 10 He was in the world , and though the world was made through him , the world did not recognize him . 11 He came to that which was his own , but his own did not receive him . 12 Yet to all who received him , to those who believed in his name , he gave the right to become children of God \u2014 13 
 children born not of natural descent , [ c ] nor of human decision or a husband 's will , but born of God . 14 The Word became flesh and made his dwelling among us . We have seen his glory , the glory of the One and Only , [ d ] who came from the Father , full of grace and truth . 15 John testifies concerning him . He cries out , saying , \u201c This was he of whom I said , \u2018 He who comes after me has surpassed me because he was before me . ' \u201d 16 From the fullness of his grace we have all received one blessing after another . 17 For the law was given through Moses ; grace and truth came through Jesus Christ . 18 No one has ever seen God , but God the One and Only , [ e][f ] who is at the Father 's side , has made him known . John the Baptist Denies Being the Christ 19 Now this was John 's testimony when the Jews of Jerusalem sent priests and Levites to ask him who he was . 20 He did not fail to confess , but confessed freely , \u201c I am not the Christ . [ g ] \u201d 21 They asked him 
 , \u201c Then who are you ? Are you Elijah ? \u201d He said , \u201c I am not . \u201d \u201c Are you the Prophet ? \u201d He answered , \u201c No. \u201d 22 Finally they said , \u201c Who are you ? Give us an answer to take back to those who sent us . What do you say about yourself ? \u201d 23 John replied in the words of Isaiah the prophet , \u201c I am the voice of one calling in the desert , \u2018 Make straight the way for the Lord . '\u201d[h ] 24 Now some Pharisees who had been sent 25 questioned him , \u201c Why then do you baptize if you are not the Christ , nor Elijah , nor the Prophet ? \u201d 26 \u201c I baptize with[i ] water , \u201d John replied , \u201c but among you stands one you do not know . 27 He is the one who comes after me , the thongs of whose sandals I am not worthy to untie . \u201d 28 This all happened at Bethany on the other side of the Jordan , where John was baptizing . Jesus the Lamb of God 29 The next day John saw Jesus coming toward him and said , \u201c Look , the Lamb of God , who takes away the sin of the
  world ! 30 This is the one I meant when I said , \u2018 A man who comes after me has surpassed me because he was before me . ' 31 I myself did not know him , but the reason I came baptizing with water was that he might be revealed to Israel . \u201d 32 Then John gave this testimony : \u201c I saw the Spirit come down from heaven as a dove and remain on him . 33 I would not have known him , except that the one who sent me to baptize with water told me , \u2018 The man on whom you see the Spirit come down and remain is he who will baptize with the Holy Spirit . ' 34 I have seen and I testify that this is the Son of God . \u201d Jesus ' First Disciples 35 The next day John was there again with two of his disciples . 36 When he saw Jesus passing by , he said , \u201c Look , the Lamb of God ! \u201d 37 When the two disciples heard him say this , they followed Jesus . 38 Turning around , Jesus saw them following and asked , \u201c What do you want ? \u201d They said , \u201c Rabbi \u201d ( which means Teacher ) , \u201c wh
 ere are you staying ? \u201d 39 \u201c Come , \u201d he replied , \u201c and you will see . \u201d So they went and saw where he was staying , and spent that day with him . It was about the tenth hour . 40 Andrew , Simon Peter 's brother , was one of the two who heard what John had said and who had followed Jesus . 41 The first thing Andrew did was to find his brother Simon and tell him , \u201c We have found the Messiah \u201d ( that is , the Christ ) . 42 And he brought him to Jesus . Jesus looked at him and said , \u201c You are Simon son of John . You will be called Cephas \u201d ( which , when translated , is Peter[j ] ) . Jesus Calls Philip and Nathanael 43 The next day Jesus decided to leave for Galilee . Finding Philip , he said to him , \u201c Follow me . \u201d 44 Philip , like Andrew and Peter , was from the town of Bethsaida . 45 Philip found Nathanael and told him , \u201c We have found the one Moses wrote about in the Law , and about whom the prophets also wrote\u2014Jesus of Nazareth , the son of Joseph . 
 \u201d 46 \u201c Nazareth ! Can anything good come from there ? \u201d Nathanael asked . \u201c Come and see , \u201d said Philip . 47 When Jesus saw Nathanael approaching , he said of him , \u201c Here is a true Israelite , in whom there is nothing false . \u201d 48 \u201c How do you know me ? \u201d Nathanael asked . Jesus answered , \u201c I saw you while you were still under the fig tree before Philip called you . \u201d 49 Then Nathanael declared , \u201c Rabbi , you are the Son of God ; you are the King of Israel . \u201d 50 Jesus said , \u201c You believe[k ] because I told you I saw you under the fig tree . You shall see greater things than that . \u201d 51 He then added , \u201c I tell you[l ] the truth , you[m ] shall see heaven open , and the angels of God ascending and descending on the Son of Man . \u201d John 2 Jesus Changes Water to Wine 1 On the third day a wedding took place at Cana in Galilee . Jesus ' mother was there , 2 and Jesus and his disciples had also been invited to the wedding . 3 When the wine was gone ,
  Jesus ' mother said to him , \u201c They have no more wine . \u201d 4 \u201c Dear woman , why do you involve me ? \u201d Jesus replied . \u201c My time has not yet come . \u201d 5 His mother said to the servants , \u201c Do whatever he tells you . \u201d 6 Nearby stood six stone water jars , the kind used by the Jews for ceremonial washing , each holding from twenty to thirty gallons . [ a ] 7 Jesus said to the servants , \u201c Fill the jars with water \u201d ; so they filled them to the brim . 8 Then he told them , \u201c Now draw some out and take it to the master of the banquet . \u201d They did so , 9 and the master of the banquet tasted the water that had been turned into wine . He did not realize where it had come from , though the servants who had drawn the water knew . Then he called the bridegroom aside 10 and said , \u201c Everyone brings out the choice wine first and then the cheaper wine after the guests have had too much to drink ; but you have saved the best till now . \u201d 11 This , the first of his mirac
 ulous signs , Jesus performed at Cana in Galilee . He thus revealed his glory , and his disciples put their faith in him . Jesus Clears the Temple 12 After this he went down to Capernaum with his mother and brothers and his disciples . There they stayed for a few days. 13 When it was almost time for the Jewish Passover , Jesus went up to Jerusalem . 14 In the temple courts he found men selling cattle , sheep and doves , and others sitting at tables exchanging money . 15 So he made a whip out of cords , and drove all from the temple area , both sheep and cattle ; he scattered the coins of the money changers and overturned their tables . 16 To those who sold doves he said , \u201c Get these out of here ! How dare you turn my Father 's house into a market ! \u201d 17 His disciples remembered that it is written : \u201c Zeal for your house will consume me . \u201d [ b ] 18 Then the Jews demanded of him , \u201c What miraculous sign can you show us to prove your authority to do all this ? \u201d 19 Jesus 
 answered them , \u201c Destroy this temple , and I will raise it again in three days. \u201d 20 The Jews replied , \u201c It has taken forty-six years to build this temple , and you are going to raise it in three days ? \u201d 21 But the temple he had spoken of was his body . 22 After he was raised from the dead , his disciples recalled what he had said . Then they believed the Scripture and the words that Jesus had spoken . 23 Now while he was in Jerusalem at the Passover Feast , many people saw the miraculous signs he was doing and believed in his name . [ c ] 24 But Jesus would not entrust himself to them , for he knew all men . 25 He did not need man 's testimony about man , for he knew what was in a man . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib2/33Bib2Jude_Epistle_1_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib2/33Bib2Jude_Epistle_1_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib2/33Bib2Jude_Epistle_1_EN.txt.txt
new file mode 100644
index 0000000..91d4589
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib2/33Bib2Jude_Epistle_1_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Jude 1 1 Jude , a servant of Jesus Christ and a brother of James , To those who have been called , who are loved by God the Father and kept by[a ] Jesus Christ : 2 Mercy , peace and love be yours in abundance . The Sin and Doom of Godless Men 3 Dear friends , although I was very eager to write to you about the salvation we share , I felt I had to write and urge you to contend for the faith that was once for all entrusted to the saints . 4 For certain men whose condemnation was written about[b ] long ago have secretly slipped in among you . They are godless men , who change the grace of our God into a license for immorality and deny Jesus Christ our only Sovereign and Lord . 5 Though you already know all this , I want to remind you that the Lord[c ] delivered his people out of Egypt , but later destroyed those who did not believe . 6 And the angels who did not keep their positions of authority but abandoned their own home\u2014these he has kept in darkness , bound with everlasting c
 hains for judgment on the great Day . 7 In a similar way , Sodom and Gomorrah and the surrounding towns gave themselves up to sexual immorality and perversion . They serve as an example of those who suffer the punishment of eternal fire . 8 In the very same way , these dreamers pollute their own bodies , reject authority and slander celestial beings . 9 But even the archangel Michael , when he was disputing with the devil about the body of Moses , did not dare to bring a slanderous accusation against him , but said , \u201c The Lord rebuke you ! \u201d 10 Yet these men speak abusively against whatever they do not understand ; and what things they do understand by instinct , like unreasoning animals\u2014these are the very things that destroy them . 11 Woe to them ! They have taken the way of Cain ; they have rushed for profit into Balaam 's error ; they have been destroyed in Korah 's rebellion . 12 These men are blemishes at your love feasts , eating with you without the slightest qualm\u2014s
 hepherds who feed only themselves . They are clouds without rain , blown along by the wind ; autumn trees , without fruit and uprooted\u2014twice dead . 13 They are wild waves of the sea , foaming up their shame ; wandering stars , for whom blackest darkness has been reserved forever . 14 Enoch , the seventh from Adam , prophesied about these men : \u201c See , the Lord is coming with thousands upon thousands of his holy ones 15 to judge everyone , and to convict all the ungodly of all the ungodly acts they have done in the ungodly way , and of all the harsh words ungodly sinners have spoken against him . \u201d 16 These men are grumblers and faultfinders ; they follow their own evil desires ; they boast about themselves and flatter others for their own advantage . A Call to Persevere 17 But , dear friends , remember what the apostles of our Lord Jesus Christ foretold . 18 They said to you , \u201c In the last times there will be scoffers who will follow their own ungodly desires . \u201d 19 These
  are the men who divide you , who follow mere natural instincts and do not have the Spirit . 20 But you , dear friends , build yourselves up in your most holy faith and pray in the Holy Spirit . 21 Keep yourselves in God 's love as you wait for the mercy of our Lord Jesus Christ to bring you to eternal life . 22 Be merciful to those who doubt ; 23 snatch others from the fire and save them ; to others show mercy , mixed with fear\u2014hating even the clothing stained by corrupted flesh . Doxology 24 To him who is able to keep you from falling and to present you before his glorious presence without fault and with great joy \u2014 25 to the only God our Savior be glory , majesty , power and authority , through Jesus Christ our Lord , before all ages , now and forevermore ! Amen . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib2/34Bib2Luke_Acts_12_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib2/34Bib2Luke_Acts_12_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib2/34Bib2Luke_Acts_12_EN.txt.txt
new file mode 100644
index 0000000..181186f
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib2/34Bib2Luke_Acts_12_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Acts 1 Jesus Taken Up Into Heaven 1 In my former book , Theophilus , I wrote about all that Jesus began to do and to teach 2 until the day he was taken up to heaven , after giving instructions through the Holy Spirit to the apostles he had chosen . 3 After his suffering , he showed himself to these men and gave many convincing proofs that he was alive . He appeared to them over a period of forty days and spoke about the kingdom of God . 4 On one occasion , while he was eating with them , he gave them this command : \u201c Do not leave Jerusalem , but wait for the gift my Father promised , which you have heard me speak about . 5 For John baptized with[a ] water , but in a few days you will be baptized with the Holy Spirit . \u201d 6 So when they met together , they asked him , \u201c Lord , are you at this time going to restore the kingdom to Israel ? \u201d 7 He said to them : \u201c It is not for you to know the times or dates the Father has set by his own authority . 8 But you will receive p
 ower when the Holy Spirit comes on you ; and you will be my witnesses in Jerusalem , and in all Judea and Samaria , and to the ends of the earth . \u201d 9 After he said this , he was taken up before their very eyes , and a cloud hid him from their sight . 10 They were looking intently up into the sky as he was going , when suddenly two men dressed in white stood beside them . 11 \u201c Men of Galilee , \u201d they said , \u201c why do you stand here looking into the sky ? This same Jesus , who has been taken from you into heaven , will come back in the same way you have seen him go into heaven . \u201d Matthias Chosen to Replace Judas 12 Then they returned to Jerusalem from the hill called the Mount of Olives , a Sabbath day\u2019s walk[b ] from the city . 13 When they arrived , they went upstairs to the room where they were staying . Those present were Peter , John , James and Andrew ; Philip and Thomas , Bartholomew and Matthew ; James son of Alphaeus and Simon the Zealot , and Judas son of James 
 . 14 They all joined together constantly in prayer , along with the women and Mary the mother of Jesus , and with his brothers . 15 In those days Peter stood up among the believers[c ] ( a group numbering about a hundred and twenty ) 16 and said , \u201c Brothers , the Scripture had to be fulfilled which the Holy Spirit spoke long ago through the mouth of David concerning Judas , who served as guide for those who arrested Jesus \u2014 17 he was one of our number and shared in this ministry . \u201d 18 ( With the reward he got for his wickedness , Judas bought a field ; there he fell headlong , his body burst open and all his intestines spilled out . 19 Everyone in Jerusalem heard about this , so they called that field in their language Akeldama , that is , Field of Blood . ) 20 \u201c For , \u201d said Peter , \u201c it is written in the book of Psalms , \u201c \u2018 May his place be deserted ; let there be no one to dwell in it , \u2019 [ d ] and , \u201c \u2018 May another take his place of leadership . \u2019 [ e 
 ] 21 Therefore it is necessary to choose one of the men who have been with us the whole time the Lord Jesus went in and out among us , 22 beginning from John\u2019s baptism to the time when Jesus was taken up from us . For one of these must become a witness with us of his resurrection . \u201d 23 So they proposed two men : Joseph called Barsabbas ( also known as Justus ) and Matthias . 24 Then they prayed , \u201c Lord , you know everyone\u2019s heart . Show us which of these two you have chosen 25 to take over this apostolic ministry , which Judas left to go where he belongs . \u201d 26 Then they cast lots , and the lot fell to Matthias ; so he was added to the eleven apostles . Acts 2 The Holy Spirit Comes at Pentecost 1 When the day of Pentecost came , they were all together in one place . 2 Suddenly a sound like the blowing of a violent wind came from heaven and filled the whole house where they were sitting . 3 They saw what seemed to be tongues of fire that separated and came to rest on each
  of them . 4 All of them were filled with the Holy Spirit and began to speak in other tongues[a ] as the Spirit enabled them . 5 Now there were staying in Jerusalem God-fearing Jews from every nation under heaven . 6 When they heard this sound , a crowd came together in bewilderment , because each one heard them speaking in his own language . 7 Utterly amazed , they asked : \u201c Are not all these men who are speaking Galileans ? 8 Then how is it that each of us hears them in his own native language ? 9 Parthians , Medes and Elamites ; residents of Mesopotamia , Judea and Cappadocia , Pontus and Asia , 10 Phrygia and Pamphylia , Egypt and the parts of Libya near Cyrene ; visitors from Rome 11 ( both Jews and converts to Judaism ) ; Cretans and Arabs\u2014we hear them declaring the wonders of God in our own tongues ! \u201d 12 Amazed and perplexed , they asked one another , \u201c What does this mean ? \u201d 13 Some , however , made fun of them and said , \u201c They have had too much wine . [ b ] 
  Peter Addresses the Crowd 14 Then Peter stood up with the Eleven , raised his voice and addressed the crowd : \u201c Fellow Jews and all of you who live in Jerusalem , let me explain this to you ; listen carefully to what I say . 15 These men are not drunk , as you suppose . It\u2019s only nine in the morning ! 16 No , this is what was spoken by the prophet Joel : 17 \u201c \u2018 In the last days , God says , I will pour out my Spirit on all people . Your sons and daughters will prophesy , your young men will see visions , your old men will dream dreams . 18 Even on my servants , both men and women , I will pour out my Spirit in those days , and they will prophesy . 19 I will show wonders in the heaven above and signs on the earth below , blood and fire and billows of smoke . 20 The sun will be turned to darkness and the moon to blood before the coming of the great and glorious day of the Lord . 21 And everyone who calls on the name of the Lord will be saved . \u2019 [ c ] 22 \u201c Men of Israel
  , listen to this : Jesus of Nazareth was a man accredited by God to you by miracles , wonders and signs , which God did among you through him , as you yourselves know . 23 This man was handed over to you by God\u2019s set purpose and foreknowledge ; and you , with the help of wicked men , [ d ] put him to death by nailing him to the cross . 24 But God raised him from the dead , freeing him from the agony of death , because it was impossible for death to keep its hold on him . 25 David said about him : \u201c \u2018 I saw the Lord always before me . Because he is at my right hand , I will not be shaken . 26 Therefore my heart is glad and my tongue rejoices ; my body also will live in hope , 27 because you will not abandon me to the grave , nor will you let your Holy One see decay . 28 You have made known to me the paths of life ; you will fill me with joy in your presence . \u2019 [ e ] 29 \u201c Brothers , I can tell you confidently that the patriarch David died and was buried , and his tomb is h
 ere to this day . 30 But he was a prophet and knew that God had promised him on oath that he would place one of his descendants on his throne . 31 Seeing what was ahead , he spoke of the resurrection of the Christ , [ f ] that he was not abandoned to the grave , nor did his body see decay . 32 God has raised this Jesus to life , and we are all witnesses of the fact . 33 Exalted to the right hand of God , he has received from the Father the promised Holy Spirit and has poured out what you now see and hear . 34 For David did not ascend to heaven , and yet he said , \u201c \u2018 The Lord said to my Lord : \u201c Sit at my right hand 35 until I make your enemies a footstool for your feet . \u201d \u2019 [ g ] 36 \u201c Therefore let all Israel be assured of this : God has made this Jesus , whom you crucified , both Lord and Christ . \u201d 37 When the people heard this , they were cut to the heart and said to Peter and the other apostles , \u201c Brothers , what shall we do ? \u201d 38 Peter replied , \u201c Repent
  and be baptized , every one of you , in the name of Jesus Christ for the forgiveness of your sins . And you will receive the gift of the Holy Spirit . 39 The promise is for you and your children and for all who are far off\u2014for all whom the Lord our God will call . \u201d 40 With many other words he warned them ; and he pleaded with them , \u201c Save yourselves from this corrupt generation . \u201d 41 Those who accepted his message were baptized , and about three thousand were added to their number that day . The Fellowship of the Believers 42 They devoted themselves to the apostles \u2019 teaching and to the fellowship , to the breaking of bread and to prayer . 43 Everyone was filled with awe , and many wonders and miraculous signs were done by the apostles . 44 All the believers were together and had everything in common . 45 Selling their possessions and goods , they gave to anyone as he had need . 46 Every day they continued to meet together in the temple courts . They broke bread in the
 ir homes and ate together with glad and sincere hearts , 47 praising God and enjoying the favor of all the people . And the Lord added to their number daily those who were being saved . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib2/35Bib2Luke_Acts_2728_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib2/35Bib2Luke_Acts_2728_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib2/35Bib2Luke_Acts_2728_EN.txt.txt
new file mode 100644
index 0000000..ef881a6
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib2/35Bib2Luke_Acts_2728_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Acts 27 Paul Sails for Rome 1 When it was decided that we would sail for Italy , Paul and some other prisoners were handed over to a centurion named Julius , who belonged to the Imperial Regiment . 2 We boarded a ship from Adramyttium about to sail for ports along the coast of the province of Asia , and we put out to sea . Aristarchus , a Macedonian from Thessalonica , was with us . 3 The next day we landed at Sidon ; and Julius , in kindness to Paul , allowed him to go to his friends so they might provide for his needs . 4 From there we put out to sea again and passed to the lee of Cyprus because the winds were against us . 5 When we had sailed across the open sea off the coast of Cilicia and Pamphylia , we landed at Myra in Lycia . 6 There the centurion found an Alexandrian ship sailing for Italy and put us on board . 7 We made slow headway for many days and had difficulty arriving off Cnidus . When the wind did not allow us to hold our course , we sailed to the lee of Crete ,
  opposite Salmone . 8 We moved along the coast with difficulty and came to a place called Fair Havens , near the town of Lasea . 9 Much time had been lost , and sailing had already become dangerous because by now it was after the Fast . [ a ] So Paul warned them , 10 \u201c Men , I can see that our voyage is going to be disastrous and bring great loss to ship and cargo , and to our own lives also . \u201d 11 But the centurion , instead of listening to what Paul said , followed the advice of the pilot and of the owner of the ship . 12 Since the harbor was unsuitable to winter in , the majority decided that we should sail on , hoping to reach Phoenix and winter there . This was a harbor in Crete , facing both southwest and northwest . The Storm 13 When a gentle south wind began to blow , they thought they had obtained what they wanted ; so they weighed anchor and sailed along the shore of Crete . 14 Before very long , a wind of hurricane force , called the \u201c northeaster , \u201d swept down f
 rom the island . 15 The ship was caught by the storm and could not head into the wind ; so we gave way to it and were driven along . 16 As we passed to the lee of a small island called Cauda , we were hardly able to make the lifeboat secure . 17 When the men had hoisted it aboard , they passed ropes under the ship itself to hold it together . Fearing that they would run aground on the sandbars of Syrtis , they lowered the sea anchor and let the ship be driven along . 18 We took such a violent battering from the storm that the next day they began to throw the cargo overboard . 19 On the third day , they threw the ship 's tackle overboard with their own hands . 20 When neither sun nor stars appeared for many days and the storm continued raging , we finally gave up all hope of being saved . 21 After the men had gone a long time without food , Paul stood up before them and said : \u201c Men , you should have taken my advice not to sail from Crete ; then you would have spared yourselves thi
 s damage and loss . 22 But now I urge you to keep up your courage , because not one of you will be lost ; only the ship will be destroyed . 23 Last night an angel of the God whose I am and whom I serve stood beside me 24 and said , \u2018 Do not be afraid , Paul . You must stand trial before Caesar ; and God has graciously given you the lives of all who sail with you . ' 25 So keep up your courage , men , for I have faith in God that it will happen just as he told me . 26 Nevertheless , we must run aground on some island . \u201d The Shipwreck 27 On the fourteenth night we were still being driven across the Adriatic[b ] Sea , when about midnight the sailors sensed they were approaching land . 28 They took soundings and found that the water was a hundred and twenty feet[c ] deep . A short time later they took soundings again and found it was ninety feet[d ] deep . 29 Fearing that we would be dashed against the rocks , they dropped four anchors from the stern and prayed for daylight . 30 In
  an attempt to escape from the ship , the sailors let the lifeboat down into the sea , pretending they were going to lower some anchors from the bow . 31 Then Paul said to the centurion and the soldiers , \u201c Unless these men stay with the ship , you cannot be saved . \u201d 32 So the soldiers cut the ropes that held the lifeboat and let it fall away . 33 Just before dawn Paul urged them all to eat . \u201c For the last fourteen days , \u201d he said , \u201c you have been in constant suspense and have gone without food\u2014you have n't eaten anything . 34 Now I urge you to take some food . You need it to survive . Not one of you will lose a single hair from his head . \u201d 35 After he said this , he took some bread and gave thanks to God in front of them all . Then he broke it and began to eat . 36 They were all encouraged and ate some food themselves . 37 Altogether there were 276 of us on board . 38 When they had eaten as much as they wanted , they lightened the ship by throwing the grain into 
 the sea . 39 When daylight came , they did not recognize the land , but they saw a bay with a sandy beach , where they decided to run the ship aground if they could . 40 Cutting loose the anchors , they left them in the sea and at the same time untied the ropes that held the rudders . Then they hoisted the foresail to the wind and made for the beach . 41 But the ship struck a sandbar and ran aground . The bow stuck fast and would not move , and the stern was broken to pieces by the pounding of the surf . 42 The soldiers planned to kill the prisoners to prevent any of them from swimming away and escaping . 43 But the centurion wanted to spare Paul 's life and kept them from carrying out their plan . He ordered those who could swim to jump overboard first and get to land . 44 The rest were to get there on planks or on pieces of the ship . In this way everyone reached land in safety . Acts 28 Ashore on Malta 1 Once safely on shore , we found out that the island was called Malta . 2 The
  islanders showed us unusual kindness . They built a fire and welcomed us all because it was raining and cold . 3 Paul gathered a pile of brushwood and , as he put it on the fire , a viper , driven out by the heat , fastened itself on his hand . 4 When the islanders saw the snake hanging from his hand , they said to each other , \u201c This man must be a murderer ; for though he escaped from the sea , Justice has not allowed him to live . \u201d 5 But Paul shook the snake off into the fire and suffered no ill effects . 6 The people expected him to swell up or suddenly fall dead , but after waiting a long time and seeing nothing unusual happen to him , they changed their minds and said he was a god . 7 There was an estate nearby that belonged to Publius , the chief official of the island . He welcomed us to his home and for three days entertained us hospitably . 8 His father was sick in bed , suffering from fever and dysentery . Paul went in to see him and , after prayer , placed his hands
  on him and healed him . 9 When this had happened , the rest of the sick on the island came and were cured . 10 They honored us in many ways and when we were ready to sail , they furnished us with the supplies we needed . Arrival at Rome 11 After three months we put out to sea in a ship that had wintered in the island . It was an Alexandrian ship with the figurehead of the twin gods Castor and Pollux . 12 We put in at Syracuse and stayed there three days. 13 From there we set sail and arrived at Rhegium . The next day the south wind came up , and on the following day we reached Puteoli . 14 There we found some brothers who invited us to spend a week with them . And so we came to Rome . 15 The brothers there had heard that we were coming , and they traveled as far as the Forum of Appius and the Three Taverns to meet us . At the sight of these men Paul thanked God and was encouraged . 16 When we got to Rome , Paul was allowed to live by himself , with a soldier to guard him . Paul Pre
 aches at Rome Under Guard 17 Three days later he called together the leaders of the Jews . When they had assembled , Paul said to them : \u201c My brothers , although I have done nothing against our people or against the customs of our ancestors , I was arrested in Jerusalem and handed over to the Romans . 18 They examined me and wanted to release me , because I was not guilty of any crime deserving death . 19 But when the Jews objected , I was compelled to appeal to Caesar\u2014not that I had any charge to bring against my own people . 20 For this reason I have asked to see you and talk with you . It is because of the hope of Israel that I am bound with this chain . \u201d 21 They replied , \u201c We have not received any letters from Judea concerning you , and none of the brothers who have come from there has reported or said anything bad about you . 22 But we want to hear what your views are , for we know that people everywhere are talking against this sect . \u201d 23 They arranged to meet Pau
 l on a certain day , and came in even larger numbers to the place where he was staying . From morning till evening he explained and declared to them the kingdom of God and tried to convince them about Jesus from the Law of Moses and from the Prophets . 24 Some were convinced by what he said , but others would not believe . 25 They disagreed among themselves and began to leave after Paul had made this final statement : \u201c The Holy Spirit spoke the truth to your forefathers when he said through Isaiah the prophet : 26 \u201c \u2018 Go to this people and say , \u201c You will be ever hearing but never understanding ; you will be ever seeing but never perceiving . \u201d 27 For this people 's heart has become calloused ; they hardly hear with their ears , and they have closed their eyes . Otherwise they might see with their eyes , hear with their ears , understand with their hearts and turn , and I would heal them . '[a ] 28 \u201c Therefore I want you to know that God 's salvation has been sent to t
 he Gentiles , and they will listen ! \u201d [ b ] 30 For two whole years Paul stayed there in his own rented house and welcomed all who came to see him . 31 Boldly and without hindrance he preached the kingdom of God and taught about the Lord Jesus Christ . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib2/36Bib2Luke_Gospel_12_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib2/36Bib2Luke_Gospel_12_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib2/36Bib2Luke_Gospel_12_EN.txt.txt
new file mode 100644
index 0000000..9a305d1
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib2/36Bib2Luke_Gospel_12_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Luke 1 Introduction 1 Many have undertaken to draw up an account of the things that have been fulfilled[a ] among us , 2 just as they were handed down to us by those who from the first were eyewitnesses and servants of the word . 3 Therefore , since I myself have carefully investigated everything from the beginning , it seemed good also to me to write an orderly account for you , most excellent Theophilus , 4 so that you may know the certainty of the things you have been taught . The Birth of John the Baptist Foretold 5 In the time of Herod king of Judea there was a priest named Zechariah , who belonged to the priestly division of Abijah ; his wife Elizabeth was also a descendant of Aaron . 6 Both of them were upright in the sight of God , observing all the Lord\u2019s commandments and regulations blamelessly . 7 But they had no children , because Elizabeth was barren ; and they were both well along in years . 8 Once when Zechariah\u2019s division was on duty and he was serving as pri
 est before God , 9 he was chosen by lot , according to the custom of the priesthood , to go into the temple of the Lord and burn incense . 10 And when the time for the burning of incense came , all the assembled worshipers were praying outside . 11 Then an angel of the Lord appeared to him , standing at the right side of the altar of incense . 12 When Zechariah saw him , he was startled and was gripped with fear . 13 But the angel said to him : \u201c Do not be afraid , Zechariah ; your prayer has been heard . Your wife Elizabeth will bear you a son , and you are to give him the name John . 14 He will be a joy and delight to you , and many will rejoice because of his birth , 15 for he will be great in the sight of the Lord . He is never to take wine or other fermented drink , and he will be filled with the Holy Spirit even from birth . [ b ] 16 Many of the people of Israel will he bring back to the Lord their God . 17 And he will go on before the Lord , in the spirit and power of Elija
 h , to turn the hearts of the fathers to their children and the disobedient to the wisdom of the righteous\u2014to make ready a people prepared for the Lord . \u201d 18 Zechariah asked the angel , \u201c How can I be sure of this ? I am an old man and my wife is well along in years . \u201d 19 The angel answered , \u201c I am Gabriel . I stand in the presence of God , and I have been sent to speak to you and to tell you this good news . 20 And now you will be silent and not able to speak until the day this happens , because you did not believe my words , which will come true at their proper time . \u201d 21 Meanwhile , the people were waiting for Zechariah and wondering why he stayed so long in the temple . 22 When he came out , he could not speak to them . They realized he had seen a vision in the temple , for he kept making signs to them but remained unable to speak . 23 When his time of service was completed , he returned home . 24 After this his wife Elizabeth became pregnant and for five months 
 remained in seclusion . 25 \u201c The Lord has done this for me , \u201d she said . \u201c In these days he has shown his favor and taken away my disgrace among the people . \u201d The Birth of Jesus Foretold 26 In the sixth month , God sent the angel Gabriel to Nazareth , a town in Galilee , 27 to a virgin pledged to be married to a man named Joseph , a descendant of David . The virgin\u2019s name was Mary . 28 The angel went to her and said , \u201c Greetings , you who are highly favored ! The Lord is with you . \u201d 29 Mary was greatly troubled at his words and wondered what kind of greeting this might be . 30 But the angel said to her , \u201c Do not be afraid , Mary , you have found favor with God . 31 You will be with child and give birth to a son , and you are to give him the name Jesus . 32 He will be great and will be called the Son of the Most High . The Lord God will give him the throne of his father David , 33 and he will reign over the house of Jacob forever ; his kingdom will never end . 
  34 \u201c How will this be , \u201d Mary asked the angel , \u201c since I am a virgin ? \u201d 35 The angel answered , \u201c The Holy Spirit will come upon you , and the power of the Most High will overshadow you . So the holy one to be born will be called[c ] the Son of God . 36 Even Elizabeth your relative is going to have a child in her old age , and she who was said to be barren is in her sixth month . 37 For nothing is impossible with God . \u201d 38 \u201c I am the Lord\u2019s servant , \u201d Mary answered . \u201c May it be to me as you have said . \u201d Then the angel left her . Mary Visits Elizabeth 39 At that time Mary got ready and hurried to a town in the hill country of Judea , 40 where she entered Zechariah\u2019s home and greeted Elizabeth . 41 When Elizabeth heard Mary\u2019s greeting , the baby leaped in her womb , and Elizabeth was filled with the Holy Spirit . 42 In a loud voice she exclaimed : \u201c Blessed are you among women , and blessed is the child you will bear ! 43 But why am I so favored , 
 that the mother of my Lord should come to me ? 44 As soon as the sound of your greeting reached my ears , the baby in my womb leaped for joy . 45 Blessed is she who has believed that what the Lord has said to her will be accomplished ! \u201d Mary\u2019s Song 46 And Mary said : \u201c My soul glorifies the Lord 47 and my spirit rejoices in God my Savior , 48 for he has been mindful of the humble state of his servant . From now on all generations will call me blessed , 49 for the Mighty One has done great things for me \u2014 holy is his name . 50 His mercy extends to those who fear him , from generation to generation . 51 He has performed mighty deeds with his arm ; he has scattered those who are proud in their inmost thoughts . 52 He has brought down rulers from their thrones but has lifted up the humble . 53 He has filled the hungry with good things but has sent the rich away empty . 54 He has helped his servant Israel , remembering to be merciful 55 to Abraham and his descendants forever , e
 ven as he said to our fathers . \u201d 56 Mary stayed with Elizabeth for about three months and then returned home . The Birth of John the Baptist 57 When it was time for Elizabeth to have her baby , she gave birth to a son . 58 Her neighbors and relatives heard that the Lord had shown her great mercy , and they shared her joy . 59 On the eighth day they came to circumcise the child , and they were going to name him after his father Zechariah , 60 but his mother spoke up and said , \u201c No ! He is to be called John . \u201d 61 They said to her , \u201c There is no one among your relatives who has that name . \u201d 62 Then they made signs to his father , to find out what he would like to name the child . 63 He asked for a writing tablet , and to everyone\u2019s astonishment he wrote , \u201c His name is John . \u201d 64 Immediately his mouth was opened and his tongue was loosed , and he began to speak , praising God . 65 The neighbors were all filled with awe , and throughout the hill country of Judea pe
 ople were talking about all these things . 66 Everyone who heard this wondered about it , asking , \u201c What then is this child going to be ? \u201d For the Lord\u2019s hand was with him . Zechariah\u2019s Song 67 His father Zechariah was filled with the Holy Spirit and prophesied : 68 \u201c Praise be to the Lord , the God of Israel , because he has come and has redeemed his people . 69 He has raised up a horn[d ] of salvation for us in the house of his servant David 70 ( as he said through his holy prophets of long ago ) , 71 salvation from our enemies and from the hand of all who hate us \u2014 72 to show mercy to our fathers and to remember his holy covenant , 73 the oath he swore to our father Abraham : 74 to rescue us from the hand of our enemies , and to enable us to serve him without fear 75 in holiness and righteousness before him all our days. 76 And you , my child , will be called a prophet of the Most High ; for you will go on before the Lord to prepare the way for him , 77 to give his 
 people the knowledge of salvation through the forgiveness of their sins , 78 because of the tender mercy of our God , by which the rising sun will come to us from heaven 79 to shine on those living in darkness and in the shadow of death , to guide our feet into the path of peace . \u201d 80 And the child grew and became strong in spirit ; and he lived in the desert until he appeared publicly to Israel . Luke 2 The Birth of Jesus 1 In those days Caesar Augustus issued a decree that a census should be taken of the entire Roman world . 2 ( This was the first census that took place while Quirinius was governor of Syria . ) 3 And everyone went to his own town to register . 4 So Joseph also went up from the town of Nazareth in Galilee to Judea , to Bethlehem the town of David , because he belonged to the house and line of David . 5 He went there to register with Mary , who was pledged to be married to him and was expecting a child . 6 While they were there , the time came for the baby to be 
 born , 7 and she gave birth to her firstborn , a son . She wrapped him in cloths and placed him in a manger , because there was no room for them in the inn . The Shepherds and the Angels 8 And there were shepherds living out in the fields nearby , keeping watch over their flocks at night . 9 An angel of the Lord appeared to them , and the glory of the Lord shone around them , and they were terrified . 10 But the angel said to them , \u201c Do not be afraid . I bring you good news of great joy that will be for all the people . 11 Today in the town of David a Savior has been born to you ; he is Christ[a ] the Lord . 12 This will be a sign to you : You will find a baby wrapped in cloths and lying in a manger . \u201d 13 Suddenly a great company of the heavenly host appeared with the angel , praising God and saying , 14 \u201c Glory to God in the highest , and on earth peace to men on whom his favor rests . \u201d 15 When the angels had left them and gone into heaven , the shepherds said to one ano
 ther , \u201c Let\u2019s go to Bethlehem and see this thing that has happened , which the Lord has told us about . \u201d 16 So they hurried off and found Mary and Joseph , and the baby , who was lying in the manger . 17 When they had seen him , they spread the word concerning what had been told them about this child , 18 and all who heard it were amazed at what the shepherds said to them . 19 But Mary treasured up all these things and pondered them in her heart . 20 The shepherds returned , glorifying and praising God for all the things they had heard and seen , which were just as they had been told . Jesus Presented in the Temple 21 On the eighth day , when it was time to circumcise him , he was named Jesus , the name the angel had given him before he had been conceived . 22 When the time of their purification according to the Law of Moses had been completed , Joseph and Mary took him to Jerusalem to present him to the Lord 23 ( as it is written in the Law of the Lord , \u201c Every firstborn
  male is to be consecrated to the Lord\u201d[b ] ) , 24 and to offer a sacrifice in keeping with what is said in the Law of the Lord : \u201c a pair of doves or two young pigeons . \u201d [ c ] 25 Now there was a man in Jerusalem called Simeon , who was righteous and devout . He was waiting for the consolation of Israel , and the Holy Spirit was upon him . 26 It had been revealed to him by the Holy Spirit that he would not die before he had seen the Lord\u2019s Christ . 27 Moved by the Spirit , he went into the temple courts . When the parents brought in the child Jesus to do for him what the custom of the Law required , 28 Simeon took him in his arms and praised God , saying : 29 \u201c Sovereign Lord , as you have promised , you now dismiss[d ] your servant in peace . 30 For my eyes have seen your salvation , 31 which you have prepared in the sight of all people , 32 a light for revelation to the Gentiles and for glory to your people Israel . \u201d 33 The child\u2019s father and mother marveled at wh
 at was said about him . 34 Then Simeon blessed them and said to Mary , his mother : \u201c This child is destined to cause the falling and rising of many in Israel , and to be a sign that will be spoken against , 35 so that the thoughts of many hearts will be revealed . And a sword will pierce your own soul too . \u201d 36 There was also a prophetess , Anna , the daughter of Phanuel , of the tribe of Asher . She was very old ; she had lived with her husband seven years after her marriage , 37 and then was a widow until she was eighty-four . [ e ] She never left the temple but worshiped night and day , fasting and praying . 38 Coming up to them at that very moment , she gave thanks to God and spoke about the child to all who were looking forward to the redemption of Jerusalem . 39 When Joseph and Mary had done everything required by the Law of the Lord , they returned to Galilee to their own town of Nazareth . 40 And the child grew and became strong ; he was filled with wisdom , and the gr
 ace of God was upon him . The Boy Jesus at the Temple 41 Every year his parents went to Jerusalem for the Feast of the Passover . 42 When he was twelve years old , they went up to the Feast , according to the custom . 43 After the Feast was over , while his parents were returning home , the boy Jesus stayed behind in Jerusalem , but they were unaware of it . 44 Thinking he was in their company , they traveled on for a day . Then they began looking for him among their relatives and friends . 45 When they did not find him , they went back to Jerusalem to look for him . 46 After three days they found him in the temple courts , sitting among the teachers , listening to them and asking them questions . 47 Everyone who heard him was amazed at his understanding and his answers . 48 When his parents saw him , they were astonished . His mother said to him , \u201c Son , why have you treated us like this ? Your father and I have been anxiously searching for you . \u201d 49 \u201c Why were you searchin
 g for me ? \u201d he asked . \u201c Didn\u2019t you know I had to be in my Father\u2019s house ? \u201d 50 But they did not understand what he was saying to them . 51 Then he went down to Nazareth with them and was obedient to them . But his mother treasured all these things in her heart . 52 And Jesus grew in wisdom and stature , and in favor with God and men . 
\ No newline at end of file


[17/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Unit/135UnitUnat_HumanRights_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Unit/135UnitUnat_HumanRights_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Unit/135UnitUnat_HumanRights_EN.txt.txt
new file mode 100644
index 0000000..834e146
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Unit/135UnitUnat_HumanRights_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff The Universal Declaration of Human Rights On December 10 , 1948 the General Assembly of the United Nations adopted and proclaimed the Universal Declaration of Human Rights the full text of which appears in the following pages . Following this historic act the Assembly called upon all Member countries to publicize the text of the Declaration and " to cause it to be disseminated , displayed , read and expounded principally in schools and other educational institutions , without distinction based on the political status of countries or territories . " PREAMBLE Whereas recognition of the inherent dignity and of the equal and inalienable rights of all members of the human family is the foundation of freedom , justice and peace in the world , Whereas disregard and contempt for human rights have resulted in barbarous acts which have outraged the conscience of mankind , and the advent of a world in which human beings shall enjoy freedom of speech and belief and freedom from fear and wan
 t has been proclaimed as the highest aspiration of the common people , Whereas it is essential , if man is not to be compelled to have recourse , as a last resort , to rebellion against tyranny and oppression , that human rights should be protected by the rule of law , Whereas it is essential to promote the development of friendly relations between nations , Whereas the peoples of the United Nations have in the Charter reaffirmed their faith in fundamental human rights , in the dignity and worth of the human person and in the equal rights of men and women and have determined to promote social progress and better standards of life in larger freedom , Whereas Member States have pledged themselves to achieve , in co-operation with the United Nations , the promotion of universal respect for and observance of human rights and fundamental freedoms , Whereas a common understanding of these rights and freedoms is of the greatest importance for the full realization of this pledge , Now , The
 refore THE GENERAL ASSEMBLY proclaims THIS UNIVERSAL DECLARATION OF HUMAN RIGHTS as a common standard of achievement for all peoples and all nations , to the end that every individual and every organ of society , keeping this Declaration constantly in mind , shall strive by teaching and education to promote respect for these rights and freedoms and by progressive measures , national and international , to secure their universal and effective recognition and observance , both among the peoples of Member States themselves and among the peoples of territories under their jurisdiction . Article 1. All human beings are born free and equal in dignity and rights . They are endowed with reason and conscience and should act towards one another in a spirit of brotherhood . Article 2. Everyone is entitled to all the rights and freedoms set forth in this Declaration , without distinction of any kind , such as race , colour , sex , language , religion , political or other opinion , national or s
 ocial origin , property , birth or other status . Furthermore , no distinction shall be made on the basis of the political , jurisdictional or international status of the country or territory to which a person belongs , whether it be independent , trust , non-self-governing or under any other limitation of sovereignty . Article 3. Everyone has the right to life , liberty and security of person . Article 4. No one shall be held in slavery or servitude ; slavery and the slave trade shall be prohibited in all their forms . Article 5. No one shall be subjected to torture or to cruel , inhuman or degrading treatment or punishment . Article 6. Everyone has the right to recognition everywhere as a person before the law . Article 7. All are equal before the law and are entitled without any discrimination to equal protection of the law . All are entitled to equal protection against any discrimination in violation of this Declaration and against any incitement to such discrimination . Article
  8. Everyone has the right to an effective remedy by the competent national tribunals for acts violating the fundamental rights granted him by the constitution or by law . Article 9. No one shall be subjected to arbitrary arrest , detention or exile . Article 10. Everyone is entitled in full equality to a fair and public hearing by an independent and impartial tribunal , in the determination of his rights and obligations and of any criminal charge against him . Article 11. ( 1 ) Everyone charged with a penal offence has the right to be presumed innocent until proved guilty according to law in a public trial at which he has had all the guarantees necessary for his defence . ( 2 ) No one shall be held guilty of any penal offence on account of any act or omission which did not constitute a penal offence , under national or international law , at the time when it was committed . Nor shall a heavier penalty be imposed than the one that was applicable at the time the penal offence was com
 mitted . Article 12. No one shall be subjected to arbitrary interference with his privacy , family , home or correspondence , nor to attacks upon his honour and reputation . Everyone has the right to the protection of the law against such interference or attacks . Article 13. ( 1 ) Everyone has the right to freedom of movement and residence within the borders of each state . ( 2 ) Everyone has the right to leave any country , including his own , and to return to his country . Article 14. ( 1 ) Everyone has the right to seek and to enjoy in other countries asylum from persecution . ( 2 ) This right may not be invoked in the case of prosecutions genuinely arising from non-political crimes or from acts contrary to the purposes and principles of the United Nations . Article 15. ( 1 ) Everyone has the right to a nationality . ( 2 ) No one shall be arbitrarily deprived of his nationality nor denied the right to change his nationality . Article 16. ( 1 ) Men and women of full age , without
  any limitation due to race , nationality or religion , have the right to marry and to found a family . They are entitled to equal rights as to marriage , during marriage and at its dissolution . ( 2 ) Marriage shall be entered into only with the free and full consent of the intending spouses . ( 3 ) The family is the natural and fundamental group unit of society and is entitled to protection by society and the State . Article 17. ( 1 ) Everyone has the right to own property alone as well as in association with others . ( 2 ) No one shall be arbitrarily deprived of his property . Article 18. Everyone has the right to freedom of thought , conscience and religion ; this right includes freedom to change his religion or belief , and freedom , either alone or in community with others and in public or private , to manifest his religion or belief in teaching , practice , worship and observance . Article 19. Everyone has the right to freedom of opinion and expression ; this right includes f
 reedom to hold opinions without interference and to seek , receive and impart information and ideas through any media and regardless of frontiers . Article 20. ( 1 ) Everyone has the right to freedom of peaceful assembly and association . ( 2 ) No one may be compelled to belong to an association . Article 21. ( 1 ) Everyone has the right to take part in the government of his country , directly or through freely chosen representatives . ( 2 ) Everyone has the right of equal access to public service in his country . ( 3 ) The will of the people shall be the basis of the authority of government ; this will shall be expressed in periodic and genuine elections which shall be by universal and equal suffrage and shall be held by secret vote or by equivalent free voting procedures . Article 22. Everyone , as a member of society , has the right to social security and is entitled to realization , through national effort and international co-operation and in accordance with the organization an
 d resources of each State , of the economic , social and cultural rights indispensable for his dignity and the free development of his personality . Article 23. ( 1 ) Everyone has the right to work , to free choice of employment , to just and favourable conditions of work and to protection against unemployment . ( 2 ) Everyone , without any discrimination , has the right to equal pay for equal work . ( 3 ) Everyone who works has the right to just and favourable remuneration ensuring for himself and his family an existence worthy of human dignity , and supplemented , if necessary , by other means of social protection . ( 4 ) Everyone has the right to form and to join trade unions for the protection of his interests . Article 24. Everyone has the right to rest and leisure , including reasonable limitation of working hours and periodic holidays with pay . Article 25. ( 1 ) Everyone has the right to a standard of living adequate for the health and well-being of himself and of his family
  , including food , clothing , housing and medical care and necessary social services , and the right to security in the event of unemployment , sickness , disability , widowhood , old age or other lack of livelihood in circumstances beyond his control . ( 2 ) Motherhood and childhood are entitled to special care and assistance . All children , whether born in or out of wedlock , shall enjoy the same social protection . Article 26. ( 1 ) Everyone has the right to education . Education shall be free , at least in the elementary and fundamental stages . Elementary education shall be compulsory . Technical and professional education shall be made generally available and higher education shall be equally accessible to all on the basis of merit . ( 2 ) Education shall be directed to the full development of the human personality and to the strengthening of respect for human rights and fundamental freedoms . It shall promote understanding , tolerance and friendship among all nations , raci
 al or religious groups , and shall further the activities of the United Nations for the maintenance of peace . ( 3 ) Parents have a prior right to choose the kind of education that shall be given to their children . Article 27. ( 1 ) Everyone has the right freely to participate in the cultural life of the community , to enjoy the arts and to share in scientific advancement and its benefits . ( 2 ) Everyone has the right to the protection of the moral and material interests resulting from any scientific , literary or artistic production of which he is the author . Article 28. Everyone is entitled to a social and international order in which the rights and freedoms set forth in this Declaration can be fully realized . Article 29. ( 1 ) Everyone has duties to the community in which alone the free and full development of his personality is possible . ( 2 ) In the exercise of his rights and freedoms , everyone shall be subject only to such limitations as are determined by law solely for 
 the purpose of securing due recognition and respect for the rights and freedoms of others and of meeting the just requirements of morality , public order and the general welfare in a democratic society . ( 3 ) These rights and freedoms may in no case be exercised contrary to the purposes and principles of the United Nations . Article 30. Nothing in this Declaration may be interpreted as implying for any State , group or person any right to engage in any activity or to perform any act aimed at the destruction of any of the rights and freedoms set forth herein . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Unit/136UnitUnat_StatuteICOJ_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Unit/136UnitUnat_StatuteICOJ_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Unit/136UnitUnat_StatuteICOJ_EN.txt.txt
new file mode 100644
index 0000000..8a6f46b
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Unit/136UnitUnat_StatuteICOJ_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff STATUTE OF THE INTERNATIONAL COURT OF JUSTICE -------------------------------------------------------------------------------- TABLE OF CONTENTS : Chapter I : Organization of the Court ( Articles 2 - 33 ) Chapter II : Competence of the Court ( Articles 34 - 38 ) Chapter III : Procedure ( Articles 39 - 64 ) Chapter IV : Advisory Opinions ( Articles 65 - 68 ) Chapter V : Amendment ( Articles 69 & 70 ) Article 1 -------------------------------------------------------------------------------- The International Court of Justice established by the Charter of the United Nations as the principal judicial organ of the United Nations shall be constituted and shall function in accordance with the provisions of the present Statute . -------------------------------------------------------------------------------- CHAPTER I - ORGANIZATION OF THE COURT -------------------------------------------------------------------------------- Article 2 The Court shall be composed of a body of independent
  judges , elected regardless of their nationality from among persons of high moral character , who possess the qualifications required in their respective countries for appointment to the highest judicial offices , or are jurisconsults of recognized competence in international law . Article 3 1. The Court shall consist of fifteen members , no two of whom may be nationals of the same state . 2. A person who for the purposes of membership in the Court could be regarded as a national of more than one state shall be deemed to be a national of the one in which he ordinarily exercises civil and political rights . Article 4 1. The members of the Court shall be elected by the General Assembly and by the Security Council from a list of persons nominated by the national groups in the Permanent Court of Arbitration , in accordance with the following provisions . 2. In the case of Members of the United Nations not represented in the Permanent Court of Arbitration , candidates shall be nominated
  by national groups appointed for this purpose by their governments under the same conditions as those prescribed for members of the Permanent Court of Arbitration by Article 44 of the Convention of The Hague of 1907 for the pacific settlement of international disputes . 3. The conditions under which a state which is a party to the present Statute but is not a Member of the United Nations may participate in electing the members of the Court shall , in the absence of a special agreement , be laid down by the General Assembly upon recommendation of the Security Council . Article 5 1. At least three months before the date of the election , the Secretary-General of the United Nations shall address a written request to the members of the Permanent Court of Arbitration belonging to the states which are parties to the present Statute , and to the members of the national groups appointed under Article 4 , paragraph 2 , inviting them to undertake , within a given time , by national groups , 
 the nomination of persons in a position to accept the duties of a member of the Court . 2. No group may nominate more than four persons , not more than two of whom shall be of their own nationality . In no case may the number of candidates nominated by a group be more than double the number of seats to be filled . Article 6 Before making these nominations , each national group is recommended to consult its highest court of justice , its legal faculties and schools of law , and its national academies and national sections of international academies devoted to the study of law . Article 7 1. The Secretary-General shall prepare a list in alphabetical order of all the persons thus nominated . Save as provided in Article 12 , paragraph 2 , these shall be the only persons eligible . 2. The Secretary-General shall submit this list to the General Assembly and to the Security Council . Article 8 The General Assembly and the Security Council shall proceed independently of one another to elect
  the members of the Court . Article 9 At every election , the electors shall bear in mind not only that the persons to be elected should individually possess the qualifications required , but also that in the body as a whole the representation of the main forms of civilization and of the principal legal systems of the world should be assured . Article 10 1. Those candidates who obtain an absolute majority of votes in the General Assembly and in the Security Council shall be considered as elected . 2. Any vote of the Security Council , whether for the election of judges or for the appointment of members of the conference envisaged in Article 12 , shall be taken without any distinction between permanent and non-permanent members of the Security Council . 3. In the event of more than one national of the same state obtaining an absolute majority of the votes both of the General Assembly and of the Security Council , the eldest of these only shall be considered as elected . Article 11 If
  , after the first meeting held for the purpose of the election , one or more seats remain to be filled , a second and , if necessary , a third meeting shall take place . Article 12 1. If , after the third meeting , one or more seats still remain unfilled , a joint conference consisting of six members , three appointed by the General Assembly and three by the Security Council , may be formed at any time at the request of either the General Assembly or the Security Council , for the purpose of choosing by the vote of an absolute majority one name for each seat still vacant , to submit to the General Assembly and the Security Council for their respective acceptance . 2. If the joint conference is unanimously agreed upon any person who fulfills the required conditions , he may be included in its list , even though he was not included in the list of nominations referred to in Article 7. 3. If the joint conference is satisfied that it will not be successful in procuring an election , tho
 se members of the Court who have already been elected shall , within a period to be fixed by the Security Council , proceed to fill the vacant seats by selection from among those candidates who have obtained votes either in the General Assembly or in the Security Council . 4. In the event of an equality of votes among the judges , the eldest judge shall have a casting vote . Article 13 1. The members of the Court shall be elected for nine years and may be re-elected ; provided , however , that of the judges elected at the first election , the terms of five judges shall expire at the end of three years and the terms of five more judges shall expire at the end of six years . 2. The judges whose terms are to expire at the end of the above-mentioned initial periods of three and six years shall be chosen by lot to be drawn by the Secretary-General immediately after the first election has been completed . 3. The members of the Court shall continue to discharge their duties until their pla
 ces have been filled . Though replaced , they shall finish any cases which they may have begun . 4. In the case of the resignation of a member of the Court , the resignation shall be addressed to the President of the Court for transmission to the Secretary-General . This last notification makes the place vacant . Article 14 Vacancies shall be filled by the same method as that laid down for the first election , subject to the following provision : the Secretary-General shall , within one month of the occurrence of the vacancy , proceed to issue the invitations provided for in Article 5 , and the date of the election shall be fixed by the Security Council . Article 15 A member of the Court elected to replace a member whose term of office has not expired shall hold office for the remainder of his predecessor 's term . Article 16 1. No member of the Court may exercise any political or administrative function , or engage in any other occupation of a professional nature . 2. Any doubt on 
 this point shall be settled by the decision of the Court . Article 17 1. No member of the Court may act as agent , counsel , or advocate in any case . 2. No member may participate in the decision of any case in which he has previously taken part as agent , counsel , or advocate for one of the parties , or as a member of a national or international court , or of a commission of enquiry , or in any other capacity . 3. Any doubt on this point shall be settled by the decision of the Court . Article 18 1. No member of the Court can be dismissed unless , in the unanimous opinion of the other members , he has ceased to fulfill the required conditions . 2. Formal notification thereof shall be made to the Secretary-General by the Registrar . 3. This notification makes the place vacant . Article 19 The members of the Court , when engaged on the business of the Court , shall enjoy diplomatic privileges and immunities . Article 20 Every member of the Court shall , before taking up his duties , 
 make a solemn declaration in open court that he will exercise his powers impartially and conscientiously . Article 21 1. The Court shall elect its President and Vice-President for three years ; they may be re-elected . 2. The Court shall appoint its Registrar and may provide for the appointment of such other officers as may be necessary . Article 22 1. The seat of the Court shall be established at The Hague . This , however , shall not prevent the Court from sitting and exercising its functions elsewhere whenever the Court considers it desirable . 2. The President and the Registrar shall reside at the seat of the Court . Article 23 1. The Court shall remain permanently in session , except during the judicial vacations , the dates and duration of which shall be fixed by the Court . 2. Members of the Court are entitled to periodic leave , the dates and duration of which shall be fixed by the Court , having in mind the distance between The Hague and the home of each judge . 3. Members 
 of the Court shall be bound , unless they are on leave or prevented from attending by illness or other serious reasons duly explained to the President , to hold themselves permanently at the disposal of the Court . Article 24 1. If , for some special reason , a member of the Court considers that he should not take part in the decision of a particular case , he shall so inform the President . 2. If the President considers that for some special reason one of the members of the Court should not sit in a particular case , he shall give him notice accordingly . 3. If in any such case the member Court and the President disagree , the matter shall be settled by the decision of the Court . Article 25 1. The full Court shall sit except when it is expressly provided otherwise in the present Statute . 2. Subject to the condition that the number of judges available to constitute the Court is not thereby reduced below eleven , the Rules of the Court may provide for allowing one or more judges , 
 according to circumstances and in rotation , to be dispensed from sitting . 3. A quorum of nine judges shall suffice to constitute the Court . Article 26 1. The Court may from time to time form one or more chambers , composed of three or more judges as the Court may determine , for dealing with particular categories of cases ; for example , labour cases and cases relating to transit and communications . 2. The Court may at any time form a chamber for dealing with a particular case . The number of judges to constitute such a chamber shall be determined by the Court with the approval of the parties . 3. Cases shall be heard and determined by the chambers provided for in this article if the parties so request . Article 27 A judgment given by any of the chambers provided for in Articles 26 and 29 shall be considered as rendered by the Court . Article 28 The chambers provided for in Articles 26 and 29 may , with the consent of the parties , sit and exercise their functions elsewhere than
  at The Hague . Article 29 With a view to the speedy dispatch of business , the Court shall form annually a chamber composed of five judges which , at the request of the parties , may hear and determine cases by summary procedure . In addition , two judges shall be selected for the purpose of replacing judges who find it impossible to sit . Article 30 1. The Court shall frame rules for carrying out its functions . In particular , it shall lay down rules of procedure . 2. The Rules of the Court may provide for assessors to sit with the Court or with any of its chambers , without the right to vote . Article 31 1. Judges of the nationality of each of the parties shall retain their right to sit in the case before the Court . 2. If the Court includes upon the Bench a judge of the nationality of one of the parties , any other party may choose a person to sit as judge . Such person shall be chosen preferably from among those persons who have been nominated as candidates as provided in Arti
 cles 4 and 5. 3. If the Court includes upon the Bench no judge of the nationality of the parties , each of these parties may proceed to choose a judge as provided in paragraph 2 of this Article . 4. The provisions of this Article shall apply to the case of Articles 26 and 29. In such cases , the President shall request one or , if necessary , two of the members of the Court forming the chamber to give place to the members of the Court of the nationality of the parties concerned , and , failing such , or if they are unable to be present , to the judges specially chosen by the parties . 5. Should there be several parties in the same interest , they shall , for the purpose of the preceding provisions , be reckoned as one party only . Any doubt upon this point shall be settled by the decision of the Court . 6. Judges chosen as laid down in paragraphs 2 , 3 , and 4 of this Article shall fulfill the conditions required by Articles 2 , 17 ( paragraph 2 ) , 20 , and 24 of the present Statut
 e . They shall take part in the decision on terms of complete equality with their colleagues . Article 32 1. Each member of the Court shall receive an annual salary . 2. The President shall receive a special annual allowance . 3. The Vice-President shall receive a special allowance for every day on which he acts as President . 4. The judges chosen under Article 31 , other than members of the Court , shall receive compensation for each day on which they exercise their functions . 5. These salaries , allowances , and compensation shall be fixed by the General Assembly . They may not be decreased during the term of office . 6. The salary of the Registrar shall be fixed by the General Assembly on the proposal of the Court . 7. Regulations made by the General Assembly shall fix the conditions under which retirement pensions may be given to members of the Court and to the Registrar , and the conditions under which members of the Court and the Registrar shall have their travelling expenses
  refunded . 8. The above salaries , allowances , and compensation shall be free of all taxation . Article 33 The expenses of the Court shall be borne by the United Nations in such a manner as shall be decided by the General Assembly . -------------------------------------------------------------------------------- CHAPTER II - COMPETENCE OF THE COURT -------------------------------------------------------------------------------- Article 34 1. Only states may be parties in cases before the Court . 2. The Court , subject to and in conformity with its Rules , may request of public international organizations information relevant to cases before it , and shall receive such information presented by such organizations on their own initiative . 3. Whenever the construction of the constituent instrument of a public international organization or of an international convention adopted thereunder is in question in a case before the Court , the Registrar shall so notify the public internationa
 l organization concerned and shall communicate to it copies of all the written proceedings . Article 35 1. The Court shall be open to the states parties to the present Statute . 2. The conditions under which the Court shall be open to other states shall , subject to the special provisions contained in treaties in force , be laid down by the Security Council , but in no case shall such conditions place the parties in a position of inequality before the Court . 3. When a state which is not a Member of the United Nations is a party to a case , the Court shall fix the amount which that party is to contribute towards the expenses of the Court . This provision shall not apply if such state is bearing a share of the expenses of the Court Article 36 1. The jurisdiction of the Court comprises all cases which the parties refer to it and all matters specially provided for in the Charter of the United Nations or in treaties and conventions in force . 2. The states parties to the present Statute
  may at any time declare that they recognize as compulsory ipso facto and without special agreement , in relation to any other state accepting the same obligation , the jurisdiction of the Court in all legal disputes concerning : a. the interpretation of a treaty ; b. any question of international law ; c. the existence of any fact which , if established , would constitute a breach of an international obligation ; d. the nature or extent of the reparation to be made for the breach of an international obligation . 3. The declarations referred to above may be made unconditionally or on condition of reciprocity on the part of several or certain states , or for a certain time . 4. Such declarations shall be deposited with the Secretary-General of the United Nations , who shall transmit copies thereof to the parties to the Statute and to the Registrar of the Court . 5. Declarations made under Article 36 of the Statute of the Permanent Court of International Justice and which are still in
  force shall be deemed , as between the parties to the present Statute , to be acceptances of the compulsory jurisdiction of the International Court of Justice for the period which they still have to run and in accordance with their terms . 6. In the event of a dispute as to whether the Court has jurisdiction , the matter shall be settled by the decision of the Court . Article 37 Whenever a treaty or convention in force provides for reference of a matter to a tribunal to have been instituted by the League of Nations , or to the Permanent Court of International Justice , the matter shall , as between the parties to the present Statute , be referred to the International Court of Justice . Article 38 1. The Court , whose function is to decide in accordance with international law such disputes as are submitted to it , shall apply : a. international conventions , whether general or particular , establishing rules expressly recognized by the contesting states ; b. international custom , a
 s evidence of a general practice accepted as law ; c. the general principles of law recognized by civilized nations ; d. subject to the provisions of Article 59 , judicial decisions and the teachings of the most highly qualified publicists of the various nations , as subsidiary means for the determination of rules of law . 2. This provision shall not prejudice the power of the Court to decide a case ex aequo et bono , if the parties agree thereto . -------------------------------------------------------------------------------- CHAPTER III - PROCEDURE -------------------------------------------------------------------------------- Article 39 1. The official languages of the Court shall be French and English . If the parties agree that the case shall be conducted in French , the judgment shall be delivered in French . If the parties agree that the case shall be conducted in English , the judgment shall be delivered in English . 2. In the absence of an agreement as to which language s
 hall be employed , each party may , in the pleadings , use the language which it prefers ; the decision of the Court shall be given in French and English . In this case the Court shall at the same time determine which of the two texts shall be considered as authoritative . 3. The Court shall , at the request of any party , authorize a language other than French or English to be used by that party . Article 40 1. Cases are brought before the Court , as the case may be , either by the notification of the special agreement or by a written application addressed to the Registrar . In either case the subject of the dispute and the parties shall be indicated . 2. The Registrar shall forthwith communicate the application to all concerned . 3. He shall also notify the Members of the United Nations through the Secretary-General , and also any other states entitled to appear before the Court . Article 41 1. The Court shall have the power to indicate , if it considers that circumstances so requ
 ire , any provisional measures which ought to be taken to preserve the respective rights of either party . 2. Pending the final decision , notice of the measures suggested shall forthwith be given to the parties and to the Security Council . Article 42 1. The parties shall be represented by agents . 2. They may have the assistance of counsel or advocates before the Court . 3. The agents , counsel , and advocates of parties before the Court shall enjoy the privileges and immunities necessary to the independent exercise of their duties . Article 43 1. The procedure shall consist of two parts : written and oral . 2. The written proceedings shall consist of the communication to the Court and to the parties of memorials , counter-memorials and , if necessary , replies ; also all papers and documents in support . 3. These communications shall be made through the Registrar , in the order and within the time fixed by the Court . 4. A certified copy of every document produced by one party sh
 all be communicated to the other party . 5. The oral proceedings shall consist of the hearing by the Court of witnesses , experts , agents , counsel , and advocates . Article 44 1. For the service of all notices upon persons other than the agents , counsel , and advocates , the Court shall apply direct to the government of the state upon whose territory the notice has to be served . 2. The same provision shall apply whenever steps are to be taken to procure evidence on the spot . Article 45 The hearing shall be under the control of the President or , if he is unable to preside , of the Vice-President ; if neither is able to preside , the senior judge present shall preside . Article 46 The hearing in Court shall be public , unless the Court shall decide otherwise , or unless the parties demand that the public be not admitted . Article 47 1. Minutes shall be made at each hearing and signed by the Registrar and the President . 2. These minutes alone shall be authentic . Article 48 The 
 Court shall make orders for the conduct of the case , shall decide the form and time in which each party must conclude its arguments , and make all arrangements connected with the taking of evidence . Article 49 The Court may , even before the hearing begins , call upon the agents to produce any document or to supply any explanations . Formal note shall be taken of any refusal . Article 50 The Court may , at any time , entrust any individual , body , bureau , commission , or other organization that it may select , with the task of carrying out an enquiry or giving an expert opinion . Article 51 During the hearing any relevant questions are to be put to the witnesses and experts under the conditions laid down by the Court in the rules of procedure referred to in Article 30. Article 52 After the Court has received the proofs and evidence within the time specified for the purpose , it may refuse to accept any further oral or written evidence that one party may desire to present unless 
 the other side consents . Article 53 1. Whenever one of the parties does not appear before the Court , or fails to defend its case , the other party may call upon the Court to decide in favour of its claim . 2. The Court must , before doing so , satisfy itself , not only that it has jurisdiction in accordance with Articles 36 and 37 , but also that the claim is well founded in fact and law . Article 54 1. When , subject to the control of the Court , the agents , counsel , and advocates have completed their presentation of the case , the President shall declare the hearing closed . 2. The Court shall withdraw to consider the judgment . 3. The deliberations of the Court shall take place in private and remain secret . Article 55 1. All questions shall be decided by a majority of the judges present . 2. In the event of an equality of votes , the President or the judge who acts in his place shall have a casting vote . Article 56 1. The judgment shall state the reasons on which it is base
 d . 2. It shall contain the names of the judges who have taken part in the decision . Article 57 If the judgment does not represent in whole or in part the unanimous opinion of the judges , any judge shall be entitled to deliver a separate opinion . Article 58 The judgment shall be signed by the President and by the Registrar . It shall be read in open court , due notice having been given to the agents . Article 59 The decision of the Court has no binding force except between the parties and in respect of that particular case . Article 60 The judgment is final and without appeal . In the event of dispute as to the meaning or scope of the judgment , the Court shall construe it upon the request of any party . Article 61 1. An application for revision of a judgment may be made only when it is based upon the discovery of some fact of such a nature as to be a decisive factor , which fact was , when the judgment was given , unknown to the Court and also to the party claiming revision , al
 ways provided that such ignorance was not due to negligence . 2. The proceedings for revision shall be opened by a judgment of the Court expressly recording the existence of the new fact , recognizing that it has such a character as to lay the case open to revision , and declaring the application admissible on this ground . 3. The Court may require previous compliance with the terms of the judgment before it admits proceedings in revision . 4. The application for revision must be made at latest within six months of the discovery of the new fact . 5. No application for revision may be made after the lapse of ten years from the date of the judgment . Article 62 l. Should a state consider that it has an interest of a legal nature which may be affected by the decision in the case , it may submit a request to the Court to be permitted to intervene . 2 It shall be for the Court to decide upon this request . Article 63 1. Whenever the construction of a convention to which states other than
  those concerned in the case are parties is in question , the Registrar shall notify all such states forthwith . 2. Every state so notified has the right to intervene in the proceedings ; but if it uses this right , the construction given by the judgment will be equally binding upon it . Article 64 Unless otherwise decided by the Court , each party shall bear its own costs . -------------------------------------------------------------------------------- CHAPTER IV - ADVISORY OPINIONS -------------------------------------------------------------------------------- Article 65 1. The Court may give an advisory opinion on any legal question at the request of whatever body may be authorized by or in accordance with the Charter of the United Nations to make such a request . 2. Questions upon which the advisory opinion of the Court is asked shall be laid before the Court by means of a written request containing an exact statement of the question upon which an opinion is required , and acc
 ompanied by all documents likely to throw light upon the question . Article 66 1. The Registrar shall forthwith give notice of the request for an advisory opinion to all states entitled to appear before the Court . 2. The Registrar shall also , by means of a special and direct communication , notify any state entitled to appear before the Court or international organization considered by the Court , or , should it not be sitting , by the President , as likely to be able to furnish information on the question , that the Court will be prepared to receive , within a time-limit to be fixed by the President , written statements , or to hear , at a public sitting to be held for the purpose , oral statements relating to the question . 3. Should any such state entitled to appear before the Court have failed to receive the special communication referred to in paragraph 2 of this Article , such state may express a desire to submit a written statement or to be heard ; and the Court will decide
  . 4. States and organizations having presented written or oral statements or both shall be permitted to comment on the statements made by other states or organizations in the form , to the extent , and within the time-limits which the Court , or , should it not be sitting , the President , shall decide in each particular case . Accordingly , the Registrar shall in due time communicate any such written statements to states and organizations having submitted similar statements . Article 67 The Court shall deliver its advisory opinions in open court , notice having been given to the Secretary-General and to the representatives of Members of the United Nations , of other states and of international organizations immediately concerned . Article 68 In the exercise of its advisory functions the Court shall further be guided by the provisions of the present Statute which apply in contentious cases to the extent to which it recognizes them to be applicable . --------------------------------
 ------------------------------------------------ CHAPTER V - AMENDMENT -------------------------------------------------------------------------------- Article 69 Amendments to the present Statute shall be effected by the same procedure as is provided by the Charter of the United Nations for amendments to that Charter , subject however to any provisions which the General Assembly upon recommendation of the Security Council may adopt concerning the participation of states which are parties to the present Statute but are not Members of the United Nations . Article 70 The Court shall have power to propose such amendments to the present Statute as it may deem necessary , through written communications to the Secretary-General , for consideration in conformity with the provisions of Article 69. -------------------------------------------------------------------------------- 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Wind/137WindParaschivoiuEt_DarrieusTurbine_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Wind/137WindParaschivoiuEt_DarrieusTurbine_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Wind/137WindParaschivoiuEt_DarrieusTurbine_EN.txt.txt
new file mode 100644
index 0000000..a403dd2
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Wind/137WindParaschivoiuEt_DarrieusTurbine_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff H-Darrieus Wind Turbine with Blade Pitch Control Abstract A procedure for computing the optimal variation of the blades ' pitch angle of an H-Darrieus wind turbine that maximizes its torque at given operational conditions is proposed and presented along with the results obtained on a 7 kW prototype . The CARDAAV code , based on the \u201c Double-Multiple Streamtube \u201d model developed by the first author , is used to determine the performances of the straight-bladed vertical axis wind turbine . This was coupled with a genetic algorithm optimizer . The azimuthal variation of the blades ' pitch angle is modeled with an analytical function whose coefficients are used as variables in the optimization process . Two types of variations were considered for the pitch angle : a simple sinusoidal one and one which is more general , relating closely the blades ' pitch to the local flow conditions along their circular path . A gain of almost 30 % in the annual energy production was obtained wi
 th the polynomial optimal pitch control . 1. Introduction Following the 1973 energy crisis , large-scale research and development programs were initiated , directed toward finding replacement solutions to the limited fossil fuel reserves . Wind energy was given , along with photovoltaic , solar , hydroelectric , biomass , and other resources , particular attention as a renewable and environmentally friendly energy alternative . Its technological progress has been spectacular , especially in the last ten years and , due to its steady growth in competitiveness , wind power developed into a mainstream energy source in many countries worldwide . At the global scale , over 74000 MW of wind power are already installed , and current estimates indicate that by 2030 wind energy could cover as much as 29 % of world\u2019s electricity needs . In the wind power domain two main technologies were considered as having the necessary potential for a viable development : the Horizontal Axis Wind Turbine
  ( HAWT ) and the Darrieus-type ( lift-based ) Vertical Axis Wind Turbine ( VAWT ) . A number of features have made HAWT to be preferred and become the dominant design type , especially in the utility-scale ( large and very large turbines ) segment . But , in certain conditions ( sites with highly turbulent wind like in the mountains or in urban environment ) , VAWTs seem to offer a better solution for the wind energy harnessing . If , through further and well-targeted research , increased attention is paid to the known VAWT drawbacks ( a somewhat less overall efficiency than the one of an HAWT , difficult/impossible self-starting , lower output due to operation closer to the ground , higher level of vibration caused by the inherent torque ripple and dynamic stall of the blades ) , at least in the \u201c small wind \u201d domain the VAWT design might become a major player . Among the most important problems that are now under study in the VAWT technology , the \u201c variable pitch \u201d for t
 he H-Darrieus turbines is regarded as a promising solution for the alleviation of the negative effects of the blades dynamic stall ( efficiency loss , vibration ) , improvement of the rotor\u2019s self starting qualities , and torque ripple smoothing [ 1\u20133 ] . At �cole Polytechnique de Montreal , Canada , in the wind energy research the major effort is devoted toward the development and improvement of the performance prediction of VAWTs [ 4 ] . The variable pitch is also included in the current research subjects in this domain , and results were already published [ 8 ] , obtained from the analysis that was carried out to determine if the performance of a VAWT , in terms of the power output , could be improved by simulating the operation of the blade just below stall . The present study aimed at determining if a more general cyclical pitch variation can be determined , so as to maximize the performance of an H-Darrieus ( straight bladed ) vertical axis wind turbine at given operation
  conditions . In order to perform this investigation , an optimization package was set up to serve in the determination of the optimal variation of the blade\u2019s pitch angle for a small two-bladed VAWT . The paper presents first the main components of the numerical tool that was developed . Then , the results of an optimization case are discussed . Finally , several conclusions are formulated at the end of the paper . 2. The Optmization Tool Since the local flow parameters on the blades vary along their circular path and differ quite significantly between the upwind and the downwind parts of the rotor , an optimization procedure had to be employed to determine the best law of variation of the blades \u2019 pitch angle . In the present study a tool for numerical optimization was set up by coupling the code CARDAAV , which computes the flow through and the performances of a VAWT , to an optimizer based on the genetic algorithm method . These ( main ) components of the optimization packag
 e are briefly presented in the following sections , along with the objective function , its variables , and the constraints that were imposed on their value during the optimization process . 2.1 . The CARDAAV Code CARDAAV , the numerical tool used in this analysis , is based on an improved version of the \u201c double-multiple streamtube \u201d ( DMS ) model [ 4 ] . This model considers a partition of the rotor in streamtubes and treats each of the two blade elements defined by a given streamtube as an actuator disk . Figure 1 illustrates such a streamtube and shows the values of the velocity of the flow at a number of key stations along it . Disk 1 represents the upwind blade element while disk 2 represents the downwind blade element . The actuator disk theory is based on the momentum conservation ; therefore , the velocities of the wind must be known in order to compute the force acting on the disks . The different values of the velocity ( see notations in Figure 1 and relations ( 1 ) d
 epend on the incoming ( \u201c undisturbed \u201d ) wind velocity and on the interference factors \U0001d42e and \U0001d42e ( 1 ) To determine the interference factors , a second set of equations is used . Those equations are derived from the blade element theory [ 4 ] , which equates , in each streamtube , the normal forces acting on the upwind and downwind blade elements to the forces acting upon the actuator disks . To compute the normal and tangential forces , the blade element theory is applied , and the lift ( \U0001d436 \U0001d459 ) and drag ( \U0001d436 \U0001d451 ) coefficients , obtained from the airfoil data , are used . For the upwind interference factor \U0001d462 , the following expression , relating it to the azimuthal angle \U0001d703 , is obtained : \U0001d462 ( \U0001d703 ) = \U0001d43e \U0001d43e( 2 ) with \U0001d43e = 8 \U0001d70b \U0001d45f ( 3 ) A similar set of equations is derived for the downwind interference factor \U0001d462 \u2032 . An interference factor equal to 1 is assumed at the beginning of the iterative process . Once the force given by the blade elem
 ent theory equates the one given by the actuator disk theory , the convergence is achieved and the upwind and downwind velocities are obtained . Then the torque and the mechanical power are computed . CARDAAV has the capability to analyze several predefined or user-defined rotor shapes with straight or curved blades ( parabola , catenary , ideal and modified troposkien , and Sandia shape ) . The code requires three main sets of input data , giving the geometry definition of the wind turbine ( diameter , height , blade section airfoil , blade shape , etc. ) , the operational conditions ( wind velocity , rotational speed , atmospheric conditions ) and the main control parameters ( convergence criterion , computation of the secondary effects , and the effect of dynamic stall ) . The software includes several dynamic stall semiempirical models : Gormont [ 5 ] and its variations ( Strickland , Paraschivoiu , and Berg ) and the one based on the indicial method [ 4 ] . Dynamic stall result
 s in increased peak aerodynamic torque and affects the structural fatigue of the Darrieus turbine . This effect significantly impacts the drive-train generator sizing and system reliability . The dynamic stall used in this study was the Berg version of the Gormont model , because it was found out to be the best correlated with the experimental studies carried out on similar rotor configurations as those used in the present investigation . CARDAAV is also able to account for the so-called \u201c secondary effects , \u201d such as those due to the rotating central tower , struts , and spoilers . CARDAAV has made it possible to design , analyze , and build more efficiently and at lower costs wind energy systems such as the Darrieus-type VAWT . The code is used to determine , at specified operational conditions , aero-dynamic forces and power output of VAWTs of any blade geometry . Wind speed can vary with height above ground according to a power law . The program output consists of the local
 -induced velocities , the local Reynolds numbers and angle of attack , the blade loads , and the azimuthal torque and power coefficient data . Each of these is calculated separately for the upwind and downwind halves of the rotor . The numerical models used by the program have been validated for different Darrieus-type VAWTs , through comparison with experimental data obtained from laboratory tests ( wind or water tunnels ) or from field tests , thus making CARDAAV a very attractive and efficient design and analysis tool . In Figure 2 the power output of the SANDIA 17 m wind turbine computed with CARDAAV is compared with experiments and results provided by other numerical codes . \U0001d434 \U0001d440 is an empirical constant used to correct \U0001d436 \U0001d459 and \U0001d436 \U0001d451 for dynamic stall effects [ 6 ] . 2.2 . The Optimizer To search for the best pitch variation law , an optimization strategy was adopted , namely , one that uses a genetic algorithm ( GA ) method . At the beginning of the optimization
  process , a genetic algorithm randomly selects an initial \u201c population \u201d composed of \u201c individuals \u201d , which are solutions of the analyzed problem computed for particular , randomly selected , values of the optimization variables . Three operations are typically performed by the genetic algorithms on the analyzed \u201c population \u201d : \u201c selection \u201d ( choice of the \u201c individuals \u201d for the next generation , according to a \u201c survival of the fittest \u201d criterion ) , \u201c crossover \u201d ( operation which allows information exchange between the \u201c individuals \u201d by swapping parts of the parameter vector in an attempt to get better \u201c individuals \u201d ) and \u201c mutation \u201d ( operation which introduces new or prematurely lost information in the form of random changes applied to randomly chosen vector components ) . Like in any optimization study , an \u201c objective function \u201d had to be defined . In this case the inverse of the rotor power , for given conditions of opera
 tion ( wind speed , rotational speed ) 1 \U0001d439 = \U0001d443 ( 4 ) was used as optimization function \U0001d439 . On the other hand , for the pitch angle the following analytical expression was considered : \U0001d70f = \U0001d465 ( 5 ) For different values given to the optimization variables \U0001d465 , the variation of the blade pitch angle \U0001d70f with the angle of azimuth \U0001d703 will be different . Since the local angle of attack \u03b1 of a blade element \U0001d6fc = s i n ( 6 ) contains the pitch angle , the later influences the aerodynamic characteristics , the torque , and ultimately the power output of the rotor . Hence , through relations ( 5 ) and ( 6 ) an indirect link is established between the objective function ( 4 ) and the optimization variables \U0001d465 , which control the variation of \U0001d70f . The genetic algorithm evolution strategy optimization package , GENIAL v1.1 [ 7 ] , was employed to minimize the objective function . This code includes three main modules , which perform the above mentioned operations . A nu
 mber of parameters are available in each module to control its functioning during the optimum search process . As mentioned above , the coefficients \U0001d465 of the pitch variation function ( 5 ) were used as optimization variables . Their values were subjected to certain constraints , to avoid any possible mathematical invalidity and to keep the pitch angle within reasonable or practical ( technically feasible ) limits . 2.3 . The Optimization Package To set up the optimization package , including GENIAL and CARDAAV as principal components , a main program ( MAIN ) and a new subroutine ( PITCH ) had to be coded . When the program is launched , MAIN reads some of the parameters that control the optimization process , namely , those that are frequently changed ( size of the \u201c population\u201d\u2014number of \u201c individuals \u201d , number of evaluations , constraints to be set on the optimization variables ) . These parameters have to be provided through the keyboard when a new optimization is 
 initiated . Then , MAIN calls the optimizer ( GENIAL ) , which takes control and carries through the optimization process . For each combination of the optimization variables , defining a distinct \u2018 individual \u201d , GENIAL calls ( using a \u201c system function \u201d ) PITCH then CARDAAV , which performs the analysis of the VAWT for that specific variation of the blades \u2019 pitch angle . With the turbine power , calculated by CARDAAV , the objective function ( 4 ) is determined and , based on it , the \u201c fitness function \u201d which characterizes that \u201c individual \u201d is obtained . Depending on its \u201c fitness \u201d , an \u201c individual \u201d can be maintained or eliminated from the \u201c population \u201d during the optimization process . Selection , crossover , and mutation are used to advance the \u201c population \u201d from one optimization cycle to the next until the optimum or the predefined limit of evolutions is reached . At the end of the optimization , MAIN outputs the optimal values of th
 e \U0001d465 variables ( as found through optimization ) , which define the best variation law for the pitch angle . Corresponding to this , the performance characteristics of the turbine are computed and stored in a file for postprocessing purposes . As the name indicates , the subroutine PITCH uses relation ( 5 ) to calculate the pitch angle over the entire circular trajectory of the blades , based on the values of \U0001d465 received from GENIAL . It stores the azimuthal and the pitch angles in a file where CARDAAV seeks this information before performing each new analysis . 3. Results This study was carried out on an H-Darrieus VAWT , having two constant-chord blades with an NACA 0015 airfoil cross section . It is a small , 7 kW rated power prototype , its rotor having the ( main ) geometrical characteristics given in Table 1. The performance ( power , power coefficient , Figures 3 and 4 ) of this turbine was computed with CARDAAV in several situations : without taking into account the infl
 uence of the dynamic stall , by applying the dynamic stall correction , with a blade pitch that had a sinusoidal variation and with a blade pitch that varied according to relation ( 5 ) in which : \U0001d465 . These calculations , as well as all the other that are presented in this paper , have been performed for a constant rotational speed of 125 rpm and at one or several different wind speed values . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Wind/138WindPurohitEt_CDMIndia_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Wind/138WindPurohitEt_CDMIndia_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Wind/138WindPurohitEt_CDMIndia_EN.txt.txt
new file mode 100644
index 0000000..44dbb9b
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Wind/138WindPurohitEt_CDMIndia_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Potential of wind power projects under the Clean Development Mechanism in India Results : CDM Potential of Wind Power Projects in India Considerable variation has been observed in the reported values of the PLF of the wind power plants in the CDM Project Design Documents ( Table 5 ) . Therefore , in this study to estimate the CDM potential of wind power projects in India the PLF of the wind power plants have been taken as 25 % . There are five regional grids within the country \u2013 the Northern , Western , Southern , Eastern and North-Eastern . Therefore , the CO2 emissions mitigation potential through wind power projects in India is estimated on the basis of the regional grids , whose emission factors have been calculated by the Central Electricity Authority ( CEA ) of the Government of India in 2006. Table 6 presents the estimated values of CDM potential through wind power projects in India on the basis of the regional baselines . We now do a sensitivity analysis with regards t
 o additionality determination . The case of lax additionality assumes that all wind power projects submitted are registered . The median case assumes that the rejection rate remains at the current level ( 2 out of 18 projects , i. e. 11 % ) . The case of stringent additionality assumes that 50 % of the projects are registered . In the lax additionality case , gross annual CER potential of wind power in India reaches 86 million . Similarly , based on the technical potential of wind power projects in India the CDM potential has been estimated as 25 million tonne . Among all the states in India , Gujarat has the largest CO2 emissions mitigation potential through wind power ( 19 million tonne ) followed by Andhra Pradesh ( 15.6 million tonne ) , Madhya Pradesh ( 10.8 million tonne ) , Karnataka ( 12.5 million tonne ) , Rajasthan ( 8.9 million tonne ) , and so on ( Table 6 ) . The annual electricity generation by wind power projects based on the gross and technical potential is also give
 n in Table 6. With 25 % PLF of wind power projects the annual gross electricity generation potential has been estimated at 99 TWh whereas the annual technical electricity generation potential has been estimated at 28 TWh . Table 7 presents the projected values of the cumulative capacity of wind power and likely CER generation using the logistic model described in the Methods section while Figure 5 shows the development over time . It may be noted that with the current trend of dissemination of wind power projects in the country , around 22 GW capacity could be installed up to the end of first crediting period in the SS scenario whereas in the OS scenario 36 GW capacity could be installed . Up to the the year 2020 , more than 44 GW capacity of the wind power projects are expected to be installed that would generate 87 million CERs . Discussion : How the CDM could be applied to the Diffusion of Wind Power Projects ? The CDM was slow to take off as after the Marrakech Accords of 2001 i
 t took another three years to define the bulk of the rules . The CDM Executive Board ( EB ) which is the body defining the CDM rules surprised many observers by taking a rigorous stance on critical issues such as baseline and additionality determination ( see below ) . Once the key rules were in place , a " gold rush " happened in 2005 and 2006. Over 1500 projects were submitted with an estimated CER volume of about 1.5 billion . However , the volume share of renewable energy projects has been less than expected due to the high attractiveness of projects reducing industrial gases and methane from waste . Out of the 1478 CDM projects submitted to the EB , 456 projects had been registered by the EB till 20th December , 2006 [ 32,33 ] . 183 CDM projects related to wind energy of which 47 have been registered , 9 requested registration and 127 were at the validation stage [ 33 ] . Figure 6 presents the status of the wind power projects from India . Out of the 89 projects submitted to th
 e UNFCCC , 18 projects had been registered and two projects had submitted the request for registration . 67 projects were at the validation stage whereas 2 projects had been rejected by the EB . The quantification of GHG benefits of a CDM project is done by means of a " baseline " . A baseline describes the ( theoretical ) emissions that would have occurred in case the CDM project was not implemented . The amounts of CERs that can be earned by the project are then calculated as the difference of baseline emissions and project emissions . The CO2 emissions mitigation benefits associated with a wind power project depend upon the amount of electricity saved . To estimate the CDM potential of wind power project in the country , the approved consolidated baseline methodology for grid-connected electricity generation from renewable sources ACM0002 ( Version 06 ) has been used . For the small scale CDM ( SSC ) projects , the small scale methodology AMS-I . D. " Grid connected renewable ele
 ctricity generation " in its version of 23rd December 2006 [ 34 ] can be used which explicitly mentions wind power for electricity generation . In India , most of the wind power projects are grid connected and substitute grid electricity . Therefore , for such systems , the baseline is the kWh produced by the renewable generating unit multiplied by an emission coefficient ( measured in g CO2eq . /kWh ) calculated in a transparent and conservative manner . This coefficient is 800 g CO2eq . /kWh for a grid where all generators use exclusively fuel oil and/or diesel fuel , whereas it is the weighted average of the so-called operating margin ( emission factor of all thermal power plants serving the grid ) and build margin ( emission factor of the most recently built plants that provide 20 % of the grid 's electricity ) . For wind power , the weight of the operating margin is 0.75 while the build margin is weighted at 0.25 . Alternatively , project developers can use the weighted average
  emissions of the current generation mix but this will always be less than the emission factor derived previously and thus unattractive . For intermittent and non-dispatchable generation types such as wind and solar photovoltaic , ACM0002 allows to weigh the operating margin ( OM ) and build margin ( BM ) at 75 % and 25 % , respectively , however , in this study we have used combines margin by using equal weights for OM and BM as given in CEA document [ 35 ] . To maintain the environmental integrity of the Kyoto Protocol , CERs are given only for " additional " activities that would otherwise not be expected to occur [ 36 ] . Therefore , any CDM project requires careful analysis of additionality . This has probably been the most contentious point in the development of the CDM and also resulted in great confusion amongst project developers [ 37,38 ] . The Kyoto Protocol stops short of requiring project proponents to show strict financial additionality \u2013 that the CDM revenue makes a
 n uneconomic project economic \u2013 and left scope for the CDM EB to refine the demonstration of additionality . The EB subsequently took a fairly strict interpretation of additionality and developed an additionality tool which formally is voluntary but which has become de facto mandatory as it was incorporated in most baseline methodologies . The additionality tool requires an investment analysis and/or a barrier analysis to determine whether the CDM project is the most attractive realistic alternative . This means that the project can be profitable and additional as long as developers can show that another project type was even more profitable . It is estimated that wind power in many countries is already competitive with fossil fuel and nuclear power if social/environmental costs are considered [ 28 ] . However , in India , in terms of costs per kWh in grid-connected areas , costs of wind power are higher than electricity provided by a coal plant projects thus be additional at any 
 rate . The unit cost of electricity generation is 0.05 \u20ac/kWh for coal and 0.06 \u20ac/kWh for fuel oil based system whereas in case of wind , the unit cost of electricity generation is 0.07 \u20ac/kWh in the best locations . The problem with this reasoning is that if wind projects are used to displace expensive grid electricity for industrial consumers ( priced at 0.09 \u20ac/kWh [ 39 ] ) , they are invariably the most attractive alternative unless they are built in locations with low wind speed . The situation for wind projects that supply to the grid at the state-guaranteed feed-in tariff is less clear ; the attractiveness depends on the level of the tariff . As the investment test will not be passed by most wind projects ( or only if they omit the tax incentives , as has been done by a project that achieved registration ) , project developers will use the barrier test . The barrier of higher capital cost compared to fossil fuel power plants is not really credible due to the abundance of
  capital for wind power in India and thus is mentioned only rarely . More credible barriers are low capacity utilization factor , and possible reduction in feed-in tariffs . The former depends on the siting of the project . The latter is very important as shown by the policy of Rajasthan ( see Table 3 ) and other states . In 2001 , Tamil Nadu Electricity Board ( TNEB ) changed its policy and froze the power purchase tariff for wind energy at Rs 2.70 per kWh with no escalation till 2006 and had informed that this power purchase tariff would be reviewed at 2006 and a new tariff would be fixed then . This was a major barrier for establishing new wind farms as other renewable energy plants continued to get a higher tariff . For instance , the power purchase tariff for electricity from an industrial waste/municipal waste based generation was Rs 3.49 for the year 2005 as against Rs 2.70 for wind energy . This policy encourages investors to invest in other renewable energy plants . Reducti
 on in power purchase tariff was a major investment barrier . Moreover , feed-in-tariffs may be replaced by the Availability Based Tariff ( ABT ) in which the generators with firm delivery of power against commitment will start getting more prices for the generated power , whereas wind power producers cannot guarantee supply of electricity and will be thus receive lower rates . For the projects that substitute grid electricity at industrial tariffs , there is the risk that the wind power benefit will melt down as liberalization permits industrial electricity consumers to choose the supplier in a competitive environment . Some projects have also highlighted the technological risks associated with new types of wind turbines . Lack of familiarity and experience with such new technologies can lead to perceptions of greater technical risk than for conventional energy sources . Doing the investment test \u2013 case study A 125 MW wind project in Karnataka calculated an IRR of 7.3 % . At that 
 rate , the project would clearly be unattractive for an investor . However , the picture changes if one analyzes the project more closely . If one uses industry averages for the investment cost ( Rs 5 crore per MW ) , the IRR is 11 % . If one includes the accelerated depreciation of 80 % in the first year and the 10 year income tax holiday , the IRR reaches 22 % ( personal communication by Mr. Sanjeev Chadha ) . It would be difficult to find serious alternatives that are more attractive . Nevertheless , the project was registered by the EB . Table 5 presents the additionality arguments of Indian wind power projects . 14 projects out of 20 have carried out investment and barrier analysis for the justification of additionality whereas 6 projects carried out the barrier analysis only . An assessment of the PDD 's indicates that the investment analysis is not convincing in most of the cases . Two wind projects from India were rejected due to lack of additionality . The rejection was mai
 nly due to the following statement in the annual report of the company that had invested in the projects : " The project is extremely beneficial on a standalone basis and has a payback period of three years with an internal rate of return in excess of 28 per cent . In addition to hedging Bajaj Auto 's power costs , this investment also provides sales tax incentives and an income tax shield " [ 40 ] . 
\ No newline at end of file


[33/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/58FictFlaubertG_Salammbo_11_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/58FictFlaubertG_Salammbo_11_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/58FictFlaubertG_Salammbo_11_EN.txt.txt
new file mode 100644
index 0000000..0cea3e9
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/58FictFlaubertG_Salammbo_11_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff CHAPTER XI IN THE TENT The man who guided Salammbo made her ascend again beyond the pharos in the direction of the Catacombs , and then go down the long suburb of Molouya , which was full of steep lanes . The sky was beginning to grow grey . Sometimes palm-wood beams jutting out from the walls obliged them to bend their heads . The two horses which were at the walk would often slip ; and thus they reached the Teveste gate . Its heavy leaves were half open ; they passed through , and it closed behind them . At first they followed the foot of the ramparts for a time , and at the height of the cisterns they took their way along the Taenia , a narrow strip of yellow earth separating the gulf from the lake and extending as far as Rhades . No one was to be seen around Carthage , whether on the sea or in the country . The slate-coloured waves chopped softly , and the light wind blowing their foam hither and thither spotted them with white rents . In spite of all her veils , Salammbo sh
 ivered in the freshness of the morning ; the motion and the open air dazed her . Then the sun rose ; it preyed on the back of her head , and she involuntarily dozed a little . The two animals rambled along side by side , their feet sinking into the silent sand . When they had passed the mountain of the Hot Springs , they went on at a more rapid rate , the ground being firmer . But although it was the season for sowing and ploughing , the fields were as empty as the desert as far as the eye could reach . Here and there were scattered heaps of corn ; at other places the barley was shedding its reddened ears . The villages showed black upon the clear horizon , with shapes incoherently carved . From time to time a half-calcined piece of wall would be found standing on the edge of the road . The roofs of the cottages were falling in , and in the interiors might be distinguished fragments of pottery , rags of clothing , and all kinds of unrecognisable utensils and broken things . Often a 
 creature clothed in tatters , with earthy face and flaming eyes would emerge from these ruins . But he would very quickly begin to run or would disappear into a hole . Salammbo and her guide did not stop . Deserted plains succeeded one another . Charcoal dust which was raised by their feet behind them , stretched in unequal trails over large spaces of perfectly white soil . Sometimes they came upon little peaceful spots , where a brook flowed amid the long grass ; and as they ascended the other bank Salammbo would pluck damp leaves to cool her hands . At the corner of a wood of rose-bays her horse shied violently at the corpse of a man which lay extended on the ground . The slave immediately settled her again on the cushions . He was one of the servants of the Temple , a man whom Schahabarim used to employ on perilous missions . With extreme precaution he now went on foot beside her and between the horses ; he would whip the animals with the end of a leathern lace wound round his ar
 m , or would perhaps take balls made of wheat , dates , and yolks of eggs wrapped in lotus leaves from a scrip hanging against his breast , and offer them to Salammbo without speaking , and running all the time . In the middle of the day three Barbarians clad in animals ' skins crossed their path . By degrees others appeared wandering in troops of ten , twelve , or twenty-five men ; many were driving goats or a limping cow . Their heavy sticks bristled with brass points ; cutlasses gleamed in their clothes , which were savagely dirty , and they opened their eyes with a look of menace and amazement . As they passed some sent them a vulgar benediction ; others obscene jests , and Schahabarim 's man replied to each in his own idiom . He told them that this was a sick youth going to be cured at a distant temple . However , the day was closing in . Barkings were heard , and they approached them . Then in the twilight they perceived an enclosure of dry stones shutting in a rambling edific
 e . A dog was running along the top of the wall . The slave threw some pebbles at him and they entered a lofty vaulted hall . A woman was crouching in the centre warming herself at a fire of brushwood , the smoke of which escaped through the holes in the ceiling . She was half hidden by her white hair which fell to her knees ; and unwilling to answer , she muttered with idiotic look words of vengeance against the Barbarians and the Carthaginians . The runner ferreted right and left . Then he returned to her and demanded something to eat . The old woman shook her head , and murmured with her eyes fixed upon the charcoal : " I was the hand . The ten fingers are cut off . The mouth eats no more . " The slave showed her a handful of gold pieces . She rushed upon them , but soon resumed her immobility . At last he placed a dagger which he had in his girdle beneath her throat . Then , trembling , she went and raised a large stone , and brought back an amphora of wine with fish from Hippo-
 Zarytus preserved in honey . Salammbo turned away from this unclean food , and fell asleep on the horses ' caparisons which were spread in a corner of the hall . He awoke her before daylight . The dog was howling . The slave went up to it quietly , and struck off its head with a single blow of his dagger . Then he rubbed the horses ' nostrils with blood to revive them . The old woman cast a malediction at him from behind . Salammbo perceived this , and pressed the amulet which she wore above her heart . They resumed their journey . From time to time she asked whether they would not arrive soon . The road undulated over little hills . Nothing was to be heard but the grating of the grasshoppers . The sun heated the yellowed grass ; the ground was all chinked with crevices which in dividing formed , as it were , monstrous paving-stones . Sometimes a viper passed , or eagles flew by ; the slave still continued running . Salammbo mused beneath her veils , and in spite of the heat did not
  lay them aside through fear of soiling her beautiful garments . At regular distances stood towers built by the Carthaginians for the purpose of keeping watch upon the tribes . They entered these for the sake of the shade , and then set out again . For prudence sake they had made a wide detour the day before . But they met with no one just now ; the region being a sterile one , the Barbarians had not passed that way . Gradually the devastation began again . Sometimes a piece of mosaic would be displayed in the centre of a field , the sole remnant of a vanished mansion ; and the leafless olive trees looked at a distance like large bushes of thorns . They passed through a town in which houses were burnt to the ground . Human skeletons might be seen along the walls . There were some , too , of dromedaries and mules . Half-gnawed carrion blocked the streets . Night fell . The sky was lowering and cloudy . They ascended again for two hours in a westerly direction , when suddenly they per
 ceived a quantity of little flames before them . These were shining at the bottom of an ampitheatre . Gold plates , as they displaced one another , glanced here and there . These were the cuirasses of the Clinabarians in the Punic camp ; then in the neighbourhood they distinguished other and more numerous lights , for the armies of the Mercenaries , now blended together , extended over a great space . Salammbo made a movement as though to advance . But Schahabarim 's man took her further away , and they passed along by the terrace which enclosed the camp of the Barbarians . A breach became visible in it , and the slave disappeared . A sentry was walking upon the top of the entrenchment with a bow in his hand and a pike on his shoulder . Salammbo drew still nearer ; the Barbarian knelt and a long arrow pierced the hem of her cloak . Then as she stood motionless and shrieking , he asked her what she wanted . " To speak to Matho , " she replied . " I am a fugitive from Carthage . " He 
 gave a whistle , which was repeated at intervals further away . Salammbo waited ; her frightened horse moved round and round , sniffing . When Matho arrived the moon was rising behind her . But she had a yellow veil with black flowers over her face , and so many draperies about her person , that it was impossible to make any guess about her . From the top of the terrace he gazed upon this vague form standing up like a phantom in the penumbrae of the evening . At last she said to him : " Lead me to your tent ! I wish it ! " A recollection which he could not define passed through his memory . He felt his heart beating . The air of command intimidated him . " Follow me ! " he said . The barrier was lowered , and immediately she was in the camp of the Barbarians . It was filled with a great tumult and a great throng . Bright fires were burning beneath hanging pots ; and their purpled reflections illuminating some places left others completely in the dark . There was shouting and calling
  ; shackled horses formed long straight lines amid the tents ; the latter were round and square , of leather or of canvas ; there were huts of reeds , and holes in the sand such as are made by dogs . Soldiers were carting faggots , resting on their elbows on the ground , or wrapping themselves up in mats and preparing to sleep ; and Salammbo 's horse sometimes stretched out a leg and jumped in order to pass over them . She remembered that she had seen them before ; but their beards were longer now , their faces still blacker , and their voices hoarser . Matho , who walked before her , waved them off with a gesture of his arm which raised his red mantle . Some kissed his hands ; others bending their spines approached him to ask for orders , for he was now veritable and sole chief of the Barbarians ; Spendius , Autaritus , and Narr ' Havas had become disheartened , and he had displayed so much audacity and obstinacy that all obeyed him . Salammbo followed him through the entire camp .
  His tent was at the end , three hundred feet from Hamilcar 's entrenchments . She noticed a wide pit on the right , and it seemed to her that faces were resting against the edge of it on a level with the ground , as decapitated heads might have done . However , their eyes moved , and from these half-opened mouths groanings escaped in the Punic tongue . Two Negroes holding resin lights stood on both sides of the door . Matho drew the canvas abruptly aside . She followed him . It was a deep tent with a pole standing up in the centre . It was lighted by a large lamp-holder shaped like a lotus and full of a yellow oil wherein floated handfuls of burning tow , and military things might be distinguished gleaming in the shade . A naked sword leaned against a stool by the side of a shield ; whips of hippopotamus leather , cymbals , bells , and necklaces were displayed pell-mell on baskets of esparto-grass ; a felt rug lay soiled with crumbs of black bread ; some copper money was carelessly
  heaped upon a round stone in a corner , and through the rents in the canvas the wind brought the dust from without , together with the smell of the elephants , which might be heard eating and shaking their chains . " Who are you ? " said Matho . She looked slowly around her without replying ; then her eyes were arrested in the background , where something bluish and sparkling fell upon a bed of palm-branches . She advanced quickly . A cry escaped her . Matho stamped his foot behind her . " Who brings you here ? why do you come ? " " To take it ! " she replied , pointing to the zaimph , and with the other hand she tore the veils from her head . He drew back with his elbows behind him , gaping , almost terrified . She felt as if she were leaning on the might of the gods ; and looking at him face to face she asked him for the zaimph ; she demanded it in words abundant and superb . Matho did not hear ; he was gazing at her , and in his eyes her garments were blended with her body . The
  clouding of the stuffs , like the splendour of her skin , was something special and belonging to her alone . Her eyes and her diamonds sparkled ; the polish of her nails continued the delicacy of the stones which loaded her fingers ; the two clasps of her tunic raised her breasts somewhat and brought them closer together , and he in thought lost himself in the narrow interval between them whence there fell a thread holding a plate of emeralds which could be seen lower down beneath the violet gauze . She had as earrings two little sapphire scales , each supporting a hollow pearl filled with liquid scent . A little drop would fall every moment through the holes in the pearl and moisten her naked shoulder . Matho watched it fall . He was carried away by ungovernable curiosity ; and , like a child laying his hand upon a strange fruit , he tremblingly and lightly touched the top of her chest with the tip of his finger : the flesh , which was somewhat cold , yielded with an elastic resis
 tance . This contact , though scarcely a sensible one , shook Matho to the very depths of his nature . An uprising of his whole being urged him towards her . He would fain have enveloped her , absorbed her , drunk her . His bosom was panting , his teeth were chattering . Taking her by the wrists he drew her gently to him , and then sat down upon a cuirass beside the palm-tree bed which was covered with a lion 's skin . She was standing . He looked up at her , holding her thus between his knees , and repeating : " How beautiful you are ! how beautiful you are ! " His eyes , which were continually fixed upon hers , pained her ; and the uncomfortableness , the repugnance increased in so acute a fashion that Salammbo put a constraint upon herself not to cry out . The thought of Schahabarim came back to her , and she resigned herself . Matho still kept her little hands in his own ; and from time to time , in spite of the priest 's command , she turned away her face and tried to thrust hi
 m off by jerking her arms . He opened his nostrils the better to breathe in the perfume which exhaled from her person . It was a fresh , indefinable emanation , which nevertheless made him dizzy , like the smoke from a perfuming-pan . She smelt of honey , pepper , incense , roses , with another odour still . But how was she thus with him in his tent , and at his disposal ? Some one no doubt had urged her . She had not come for the zaimph . His arms fell , and he bent his head whelmed in sudden reverie . To soften him Salammbo said to him in a plaintive voice : " What have I done to you that you should desire my death ? " " Your death ! " She resumed : " I saw you one evening by the light of my burning gardens amid fuming cups and my slaughtered slaves , and your anger was so strong that you bounded towards me and I was obliged to fly ! Then terror entered into Carthage . There were cries of the devastation of the towns , the burning of the country-seats , the massacre of the soldier
 y ; it was you who had ruined them , it was you who had murdered them ! I hate you ! Your very name gnaws me like remorse ! You are execrated more than the plague , and the Roman war ! The provinces shudder at your fury , the furrows are full of corpses ! I have followed the traces of your fires as though I were travelling behind Moloch ! " Matho leaped up ; his heart was swelling with colossal pride ; he was raised to the stature of a god . With quivering nostrils and clenched teeth she went on : " As if your sacrilege were not enough , you came to me in my sleep covered with the zaimph ! Your words I did not understand ; but I could see that you wished to drag me to some terrible thing at the bottom of an abyss . " Matho , writhing his arms , exclaimed : " No ! no ! it was to give it to you ! to restore it to you ! It seemed to me that the goddess had left her garment for you , and that it belonged to you ! In her temple or in your house , what does it matter ? are you not all-pow
 erful , immaculate , radiant and beautiful even as Tanith ? " And with a look of boundless adoration he added : " Unless perhaps you are Tanith ? " " I , Tanith ! " said Salammbo to herself . They left off speaking . The thunder rolled in the distance . Some sheep bleated , frightened by the storm . " Oh ! come near ! " he went on , " come near ! fear nothing ! " Formerly I was only a soldier mingled with the common herd of the Mercenaries , ay , and so meek that I used to carry wood on my back for the others . Do I trouble myself about Carthage ! The crowd of its people move as though lost in the dust of your sandals , and all its treasures , with the provinces , fleets , and islands , do not raise my envy like the freshness of your lips and the turn of your shoulders . But I wanted to throw down its walls that I might reach you to possess you ! Moreover , I was revenging myself in the meantime ! At present I crush men like shells , and I throw myself upon phalanxes ; I put aside t
 he sarissae with my hands , I check the stallions by the nostrils ; a catapult would not kill me ! Oh ! if you knew how I think of you in the midst of war ! Sometimes the memory of a gesture or of a fold of your garment suddenly seizes me and entwines me like a net ! I perceive your eyes in the flames of the phalaricas and on the gilding of the shields ! I hear your voice in the sounding of the cymbals . I turn aside , but you are not there ! and I plunge again into the battle ! " He raised his arms whereon his veins crossed one another like ivy on the branches of a tree . Sweat flowed down his breast between his square muscles ; and his breathing shook his sides with his bronze girdle all garnished with thongs hanging down to his knees , which were firmer than marble . Salammbo , who was accustomed to eunuchs , yielded to amazement at the strength of this man . It was the chastisement of the goddess or the influence of Moloch in motion around her in the five armies . She was overwh
 elmed with lassitude ; and she listened in a state of stupor to the intermittent shouts of the sentinels as they answered one another . The flames of the lamp kindled in the squalls of hot air . There came at times broad lightning flashes ; then the darkness increased ; and she could only see Matho 's eyeballs like two coals in the night . However , she felt that a fatality was surrounding her , that she had reached a supreme and irrevocable moment , and making an effort she went up again towards the zaimph and raised her hands to seize it . " What are you doing ? " exclaimed Matho . " I am going back to Carthage , " she placidly replied . He advanced folding his arms and with so terrible a look that her heels were immediately nailed , as it were , to the spot . " Going back to Carthage ! " He stammered , and , grinding his teeth , repeated : " Going back to Carthage ! Ah ! you came to take the zaimph , to conquer me , and then disappear ! No , no ! you belong to me ! and no one now
  shall tear you from here ! Oh ! I have not forgotten the insolence of your large tranquil eyes , and how you crushed me with the haughtiness of your beauty ! 'Tis my turn now ! You are my captive , my slave , my servant ! Call , if you like , on your father and his army , the Ancients , the rich , and your whole accursed people ! I am the master of three hundred thousand soldiers ! I will go and seek them in Lusitania , in the Gauls , and in the depths of the desert , and I will overthrow your town and burn all its temples ; the triremes shall float on the waves of blood ! I will not have a house , a stone , or a palm tree remaining ! And if men fail me I will draw the bears from the mountains and urge on the lions ! Seek not to fly or I kill you ! " Pale and with clenched fists he quivered like a harp whose strings are about to burst . Suddenly sobs stifled him , and he sank down upon his hams . " Ah ! forgive me ! I am a scoundrel , and viler than scorpions , than mire and dust !
  Just now while you were speaking your breath passed across my face , and I rejoiced like a dying man who drinks lying flat on the edge of a stream . Crush me , if only I feel your feet ! curse me , if only I hear your voice ! Do not go ! have pity ! I love you ! I love you ! " He was on his knees on the ground before her ; and he encircled her form with both his arms , his head thrown back , and his hands wandering ; the gold discs hanging from his ears gleamed upon his bronzed neck ; big tears rolled in his eyes like silver globes ; he sighed caressingly , and murmured vague words lighter than a breeze and sweet as a kiss . Salammbo was invaded by a weakness in which she lost all consciousness of herself . Something at once inward and lofty , a command from the gods , obliged her to yield herself ; clouds uplifted her , and she fell back swooning upon the bed amid the lion 's hair . The zaimph fell , and enveloped her ; she could see Matho 's face bending down above her breast . "
  Moloch , thou burnest me ! " and the soldier 's kisses , more devouring than flames , covered her ; she was as though swept away in a hurricane , taken in the might of the sun . He kissed all her fingers , her arms , her feet , and the long tresses of her hair from one end to the other . " Carry it off , " he said , " what do I care ? take me away with it ! I abandon the army ! I renounce everything ! Beyond Gades , twenty days ' journey into the sea , you come to an island covered with gold dust , verdure , and birds . On the mountains large flowers filled with smoking perfumes rock like eternal censers ; in the citron trees , which are higher than cedars , milk-coloured serpents cause the fruit to fall upon the turf with the diamonds in their jaws ; the air is so mild that it keeps you from dying . Oh ! I shall find it , you will see . We shall live in crystal grottoes cut out at the foot of the hills . No one dwells in it yet , or I shall become the king of the country . " He br
 ushed the dust off her cothurni ; he wanted her to put a quarter of a pomegranate between her lips ; he heaped up garments behind her head to make a cushion for her . He sought for means to serve her , and to humble himself , and he even spread the zaimph over her feet as if it were a mere rug . " Have you still , " he said , " those little gazelle 's horns on which your necklaces hang ? You will give them to me ! I love them ! " For he spoke as if the war were finished , and joyful laughs broke from him . The Mercenaries , Hamilcar , every obstacle had now disappeared . The moon was gliding between two clouds . They could see it through an opening in the tent . " Ah , what nights have I spent gazing at her ! she seemed to me like a veil that hid your face ; you would look at me through her ; the memory of you was mingled with her beams ; then I could no longer distinguish you ! " And with his head between her breasts he wept copiously . " And this , " she thought , " is the formida
 ble man who makes Carthage tremble ! " He fell asleep . Then disengaging herself from his arm she put one foot to the ground , and she perceived that her chainlet was broken . The maidens of the great families were accustomed to respect these shackles as something that was almost religious , and Salammbo , blushing , rolled the two pieces of the golden chain around her ankles . Carthage , Megara , her house , her room , and the country that she had passed through , whirled in tumultuous yet distinct images through her memory . But an abyss had yawned and thrown them far back to an infinite distance from her . The storm was departing ; drops of water splashing rarely , one by one , made the tent-roof shake . Matho slept like a drunken man , stretched on his side , and with one arm over the edge of the couch . His band of pearls was raised somewhat , and uncovered his brow ; his teeth were parted in a smile ; they shone through his black beard , and there was a silent and almost outra
 geous gaiety in his half-closed eyelids . Salammbo looked at him motionless , her head bent and her hands crossed . A dagger was displayed on the table of cypress-wood at the head of the bed ; the sight of the gleaming blade fired her with a sanguinary desire . Mournful voices lingered at a distance in the shade , and like a chorus of geniuses urged her on . She approached it ; she seized the steel by the handle . At the rustling of her dress Matho half opened his eyes , putting forth his mouth upon her hands , and the dagger fell . Shouts arose ; a terrible light flashed behind the canvas . Matho raised the latter ; they perceived the camp of the Libyans enveloped in great flames . Their reed huts were burning , and the twisting stems burst in the smoke and flew off like arrows ; black shadows ran about distractedly on the red horizon . They could hear the shrieks of those who were in the huts ; the elephants , oxen , and horses plunged in the midst of the crowd crushing it togethe
 r with the stores and baggage that were being rescued from the fire . Trumpets sounded . There were calls of " Matho ! Matho ! " Some people at the door tried to get in . " Come along ! Hamilcar is burning the camp of Autaritus ! " He made a spring . She found herself quite alone . Then she examined the zaimph ; and when she had viewed it well she was surprised that she had not the happiness which she had once imagined to herself . She stood with melancholy before her accomplished dream . But the lower part of the tent was raised , and a monstrous form appeared . Salammbo could at first distinguish only the two eyes and a long white beard which hung down to the ground ; for the rest of the body , which was cumbered with the rags of a tawny garment , trailed along the earth ; and with every forward movement the hands passed into the beard and then fell again . Crawling in this way it reached her feet , and Salammbo recognised the aged Gisco . In fact , the Mercenaries had broken the 
 legs of the captive Ancients with a brass bar to prevent them from taking to flight ; and they were all rotting pell-mell in a pit in the midst of filth . But the sturdiest of them raised themselves and shouted when they heard the noise of platters , and it was in this way that Gisco had seen Salammbo . He had guessed that she was a Carthaginian woman by the little balls of sandastrum flapping against her cothurni ; and having a presentiment of an important mystery he had succeeded , with the assistance of his companions , in getting out of the pit ; then with elbows and hands he had dragged himself twenty paces further on as far as Matho 's tent . Two voices were speaking within it . He had listened outside and had heard everything . " It is you ! " she said at last , almost terrified . " Yes , it is I ! " he replied , raising himself on his wrists . " They think me dead , do they not ? " She bent her head . He resumed : " Ah ! why have the Baals not granted me this mercy ! " He ap
 proached so close he was touching her . " They would have spared me the pain of cursing you ! " Salammbo sprang quickly back , so much afraid was she of this unclean being , who was as hideous as a larva and nearly as terrible as a phantom . " I am nearly one hundred years old , " he said . " I have seen Agathocles ; I have seen Regulus and the eagles of the Romans passing over the harvests of the Punic fields ! I have seen all the terrors of battles and the sea encumbered with the wrecks of our fleets ! Barbarians whom I used to command have chained my four limbs like a slave that has committed murder . My companions are dying around me , one after the other ; the odour of their corpses awakes me in the night ; I drive away the birds that come to peck out their eyes ; and yet not for a single day have I despaired of Carthage ! Though I had seen all the armies of the earth against her , and the flames of the siege overtop the height of the temples , I should have still believed in h
 er eternity ! But now all is over ! all is lost ! The gods execrate her ! A curse upon you who have quickened her ruin by your disgrace ! " She opened her lips . " Ah ! I was there ! " he cried . " I heard you gurgling with love like a prostitute ; then he told you of his desire , and you allowed him to kiss your hands ! But if the frenzy of your unchastity urged you to it , you should at least have done as do the fallow deer , which hide themselves in their copulations , and not have displayed your shame beneath your father 's very eyes ! " " What ? " she said . " Ah ! you did not know that the two entrenchments are sixty cubits from each other and that your Matho , in the excess of his pride , has posted himself just in front of Hamilcar . Your father is there behind you ; and could I climb the path which leads to the platform , I should cry to him : 'Come and see your daughter in the Barbarian 's arms ! She has put on the garment of the goddess to please him ; and in yielding her
  body to him she surrenders with the glory of your name the majesty of the gods , the vengeance of her country , even the safety of Carthage ! ' " The motion of his toothless mouth moved his beard throughout its length ; his eyes were riveted upon her and devoured her ; panting in the dust he repeated : " Ah ! sacrilegious one ! May you be accursed ! accursed ! accursed ! " Salammbo had drawn back the canvas ; she held it raised at arm 's length , and without answering him she looked in the direction of Hamilcar . " It is this way , is it not ? " she said . " What matters it to you ? Turn away ! Begone ! Rather crush your face against the earth ! It is a holy spot which would be polluted by your gaze ! " She threw the zaimph about her waist , and quickly picked up her veils , mantle , and scarf . " I hasten thither ! " she cried ; and making her escape Salammbo disappeared . At first she walked through the darkness without meeting any one , for all were betaking themselves to the fi
 re ; the uproar was increasing and great flames purpled the sky behind ; a long terrace stopped her . She turned round to right and left at random , seeking for a ladder , a rope , a stone , something in short to assist her . She was afraid of Gisco , and it seemed to her that shouts and footsteps were pursuing her . Day was beginning to break . She perceived a path in the thickness of the entrenchment . She took the hem of her robe , which impeded her , in her teeth , and in three bounds she was on the platform . A sonorous shout burst forth beneath her in the shade , the same which she had heard at the foot of the galley staircase , and leaning over she recognised Schahabarim 's man with his coupled horses . He had wandered all night between the two entrenchments ; then disquieted by the fire , he had gone back again trying to see what was passing in Matho 's camp ; and , knowing that this spot was nearest to his tent , he had not stirred from it , in obedience to the priest 's co
 mmand . He stood up on one of the horses . Salammbo let herself slide down to him ; and they fled at full gallop , circling the Punic camp in search of a gate . Matho had re-entered his tent . The smoky lamp gave but little light , and he also believed that Salammbo was asleep . Then he delicately touched the lion 's skin on the palm-tree bed . He called but she did not answer ; he quickly tore away a strip of the canvas to let in some light ; the zaimph was gone . The earth trembled beneath thronging feet . Shouts , neighings , and clashing of armour rose in the air , and clarion flourishes sounded the charge . It was as though a hurricane were whirling around him . Immoderate frenzy made him leap upon his arms , and he dashed outside . The long files of the Barbarians were descending the mountain at a run , and the Punic squares were advancing against them with a heavy and regular oscillation . The mist , rent by the rays of the sun , formed little rocking clouds which as they ros
 e gradually discovered standards , helmets , and points of pikes . Beneath the rapid evolutions portions of the earth which were still in the shadow seemed to be displaced bodily ; in other places it looked as if huge torrents were crossing one another , while thorny masses stood motionless between them . Matho could distinguish the captains , soldiers , heralds , and even the serving-men , who were mounted on asses in the rear . But instead of maintaining his position in order to cover the foot-soldiers , Narr ' Havas turned abruptly to the right , as though he wished himself to be crushed by Hamilcar . His horsemen outstripped the elephants , which were slackening their speed ; and all the horses , stretching out their unbridled heads , galloped at so furious a rate that their bellies seemed to graze the earth . Then suddenly Narr ' Havas went resolutely up to a sentry . He threw away his sword , lance , and javelins , and disappeared among the Carthaginians . The king of the Numi
 dians reached Hamilcar 's tent , and pointing to his men , who were standing still at a distance , he said : " Barca ! I bring them to you . They are yours . " Then he prostrated himself in token of bondage , and to prove his fidelity recalled all his conduct from the beginning of the war . First , he had prevented the siege of Carthage and the massacre of the captives ; then he had taken no advantage of the victory over Hanno after the defeat at Utica . As to the Tyrian towns , they were on the frontiers of his kingdom . Finally he had not taken part in the battle of the Macaras ; and he had even expressly absented himself in order to evade the obligation of fighting against the Suffet . Narr ' Havas had in fact wished to aggrandise himself by encroachments upon the Punic provinces , and had alternately assisted and forsaken the Mercenaries according to the chances of victory . But seeing that Hamilcar would ultimately prove the stronger , he had gone over to him ; and in his deser
 tion there was perhaps something of a grudge against Matho , whether on account of the command or of his former love . The Suffet listened without interrupting him . The man who thus presented himself with an army where vengeance was his due was not an auxiliary to be despised ; Hamilcar at once divined the utility of such an alliance in his great projects . With the Numidians he would get rid of the Libyans . Then he would draw off the West to the conquest of Iberia ; and , without asking Narr ' Havas why he had not come sooner , or noticing any of his lies , he kissed him , striking his breast thrice against his own . It was to bring matters to an end and in despair that he had fired the camp of the Libyans . This army came to him like a relief from the gods ; dissembling his joy he replied : " May the Baals favour you ! I do not know what the Republic will do for you , but Hamilcar is not ungrateful . " The tumult increased ; some captains entered . He was arming himself as he sp
 oke . " Come , return ! You will use your horsemen to beat down their infantry between your elephants and mine . Courage ! exterminate them ! " And Narr ' Havas was rushing away when Salammbo appeared . She leaped down quickly from her horse . She opened her ample cloak and spreading out her arms displayed the zaimph . The leathern tent , which was raised at the corners , left visible the entire circuit of the mountain with its thronging soldiers , and as it was in the centre Salammbo could be seen on all sides . An immense shouting burst forth , a long cry of triumph and hope . Those who were marching stopped ; the dying leaned on their elbows and turned round to bless her . All the Barbarians knew now that she had recovered the zaimph ; they saw her or believed that they saw her from a distance ; and other cries , but those of rage and vengeance , resounded in spite of the plaudits of the Carthaginians . Thus did the five armies in tiers upon the mountain stamp and shriek around S
 alammbo . Hamilcar , who was unable to speak , nodded her his thanks . His eyes were directed alternately upon the zaimph and upon her , and he noticed that her chainlet was broken . Then he shivered , being seized with a terrible suspicion . But soon recovering his impassibility he looked sideways at Narr ' Havas without turning his face . The king of the Numidians held himself apart in a discreet attitude ; on his forehead he bore a little of the dust which he had touched when prostrating himself . At last the Suffet advanced towards him with a look full of gravity . " As a reward for the services which you have rendered me , Narr ' Havas , I give you my daughter . Be my son , " he added , " and defend your father ! " Narr ' Havas gave a great gesture of surprise ; then he threw himself upon Hamilcar 's hands and covered them with kisses . Salammbo , calm as a statue , did not seem to understand . She blushed a little as she cast down her eyelids , and her long curved lashes made 
 shadows upon her cheeks . Hamilcar wished to unite them immediately in indissoluble betrothal . A lance was placed in Salammbo 's hands and by her offered to Narr ' Havas ; their thumbs were tied together with a thong of ox-leather ; then corn was poured upon their heads , and the grains that fell around them rang like rebounding hail . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/59FictFlaubertG_Salammbo_2_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/59FictFlaubertG_Salammbo_2_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/59FictFlaubertG_Salammbo_2_EN.txt.txt
new file mode 100644
index 0000000..ed019e3
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/59FictFlaubertG_Salammbo_2_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff CHAPTER II AT SICCA Two days afterwards the Mercenaries left Carthage . They had each received a piece of gold on the condition that they should go into camp at Sicca , and they had been told with all sorts of caresses : " You are the saviours of Carthage ! But you would starve it if you remained there ; it would become insolvent . Withdraw ! The Republic will be grateful to you later for all this condescension . We are going to levy taxes immediately ; your pay shall be in full , and galleys shall be equipped to take you back to your native lands . " They did not know how to reply to all this talk . These men , accustomed as they were to war , were wearied by residence in a town ; there was difficulty in convincing them , and the people mounted the walls to see them go away . They defiled through the street of Khamon , and the Cirta gate , pell-mell , archers with hoplites , captains with soldiers , Lusitanians with Greeks . They marched with a bold step , rattling their heavy 
 cothurni on the paving stones . Their armour was dented by the catapult , and their faces blackened by the sunburn of battles . Hoarse cries issued from their thick bears , their tattered coats of mail flapped upon the pommels of their swords , and through the holes in the brass might be seen their naked limbs , as frightful as engines of war . Sarissae , axes , spears , felt caps and bronze helmets , all swung together with a single motion . They filled the street thickly enough to have made the walls crack , and the long mass of armed soldiers overflowed between the lofty bitumen-smeared houses six storys high . Behind their gratings of iron or reed the women , with veiled heads , silently watched the Barbarians pass . The terraces , fortifications , and walls were hidden beneath the crowd of Carthaginians , who were dressed in garments of black . The sailors ' tunics showed like drops of blood among the dark multitude , and nearly naked children , whose skin shone beneath their c
 opper bracelets , gesticulated in the foliage of the columns , or amid the branches of a palm tree . Some of the Ancients were posted on the platform of the towers , and people did not know why a personage with a long beard stood thus in a dreamy attitude here and there . He appeared in the distance against the background of the sky , vague as a phantom and motionless as stone . All , however , were oppressed with the same anxiety ; it was feared that the Barbarians , seeing themselves so strong , might take a fancy to stay . But they were leaving with so much good faith that the Carthaginians grew bold and mingled with the soldiers . They overwhelmed them with protestations and embraces . Some with exaggerated politeness and audacious hypocrisy even sought to induce them not to leave the city . They threw perfumes , flowers , and pieces of silver to them . They gave them amulets to avert sickness ; but they had spit upon them three times to attract death , or had enclosed jackal 's
  hair within them to put cowardice into their hearts . Aloud , they invoked Melkarth 's favour , and in a whisper , his curse . Then came the mob of baggage , beasts of burden , and stragglers . The sick groaned on the backs of dromedaries , while others limped along leaning on broken pikes . The drunkards carried leathern bottles , and the greedy quarters of meat , cakes , fruits , butter wrapped in fig leaves , and snow in linen bags . Some were to be seen with parasols in their hands , and parrots on their shoulders . They had mastiffs , gazelles , and panthers following behind them . Women of Libyan race , mounted on asses , inveighed against the Negresses who had forsaken the lupanaria of Malqua for the soldiers ; many of them were suckling children suspended on their bosoms by leathern thongs . The mules were goaded out at the point of the sword , their backs bending beneath the load of tents , while there were numbers of serving-men and water-carriers , emaciated , jaundiced 
 with fever , and filthy with vermin , the scum of the Carthaginian populace , who had attached themselves to the Barbarians . When they had passed , the gates were shut behind them , but the people did not descend from the walls . The army soon spread over the breadth of the isthmus . It parted into unequal masses . Then the lances appeared like tall blades of grass , and finally all was lost in a train of dust ; those of the soldiers who looked back towards Carthage could now only see its long walls with their vacant battlements cut out against the edge of the sky . Then the Barbarians heard a great shout . They thought that some from among them ( for they did not know their own number ) had remained in the town , and were amusing themselves by pillaging a temple . They laughed a great deal at the idea of this , and then continued their journey . They were rejoiced to find themselves , as in former days , marching all together in the open country , and some of the Greeks sang the o
 ld song of the Mamertines : " With my lance and sword I plough and reap ; I am master of the house ! The disarmed man falls at my feet and calls me Lord and Great King . " They shouted , they leaped , the merriest began to tell stories ; the time of their miseries was past . As they arrived at Tunis , some of them remarked that a troop of Balearic slingers was missing . They were doubtless not far off ; and no further heed was paid to them . Some went to lodge in the houses , others camped at the foot of the walls , and the townspeople came out to chat with the soldiers . During the whole night fires were seen burning on the horizon in the direction of Carthage ; the light stretched like giant torches across the motionless lake . No one in the army could tell what festival was being celebrated . On the following day the Barbarian 's passed through a region that was covered with cultivation . The domains of the patricians succeeded one another along the border of the route ; channels
  of water flowed through woods of palm ; there were long , green lines of olive-trees ; rose-coloured vapours floated in the gorges of the hills , while blue mountains reared themselves behind . A warm wind was blowing . Chameleons were crawling on the broad leaves of the cactus . The Barbarians slackened their speed . They marched on in isolated detachments , or lagged behind one another at long intervals . They ate grapes along the margin of the vines . They lay on the grass and gazed with stupefaction upon the large , artificially twisted horns of the oxen , the sheep clothed with skins to protect their wool , the furrows crossing one another so as to form lozenges , and the ploughshares like ships ' anchors , with the pomegranate trees that were watered with silphium . Such wealth of the soil and such inventions of wisdom dazzled them . In the evening they stretched themselves on the tents without unfolding them ; and thought with regret of Hamilcar 's feast , as they fell aslee
 p with their faces towards the stars . In the middle of the following day they halted on the bank of a river , amid clumps of rose-bays . Then they quickly threw aside lances , bucklers and belts . They bathed with shouts , and drew water in their helmets , while others drank lying flat on their stomachs , and all in the midst of the beasts of burden whose baggage was slipping from them . Spendius , who was seated on a dromedary stolen in Hamilcar 's parks , perceived Matho at a distance , with his arm hanging against his breast , his head bare , and his face bent down , giving his mule drink , and watching the water flow . Spendius immediately ran through the crowd calling him , " Master ! master ! " Matho gave him but scant thanks for his blessings , but Spendius paid no heed to this , and began to march behind him , from time to time turning restless glances in the direction of Carthage . He was the son of a Greek rhetor and a Campanian prostitute . He had at first grown rich by 
 dealing in women ; then , ruined by a shipwreck , he had made war against the Romans with the herdsmen of Samnium . He had been taken and had escaped ; he had been retaken , and had worked in the quarries , panted in the vapour-baths , shrieked under torture , passed through the hands of many masters , and experienced every frenzy . At last , one day , in despair , he had flung himself into the sea from the top of a trireme where he was working at the oar . Some of Hamilcar 's sailors had picked him up when at the point of death , and had brought him to the ergastulum of Megara , at Carthage . But , as fugitives were to be given back to the Romans , he had taken advantage of the confusion to fly with the soldiers . During the whole of the march he remained near Matho ; he brought him food , assisted him to dismount , and spread a carpet in the evening beneath his head . Matho at last was touched by these attentions , and by degrees unlocked his lips . He had been born in the gulf of
  Syrtis . His father had taken him on a pilgrimage to the temple of Ammon . Then he had hunted elephants in the forests of the Garamantes . Afterwards he had entered the service of Carthage . He had been appointed tetrarch at the capture of Drepanum . The Republic owed him four horses , twenty-three medimni of wheat , and a winter 's pay . He feared the gods , and wished to die in his native land . Spendius spoke to him of his travels , and of the peoples and temples that he had visited . He knew many things : he could make sandals , boar-spears and nets ; he could tame wild beasts and could cook fish . Sometimes he would interrupt himself , and utter a hoarse cry from the depths of his throat ; Matho 's mule would quicken his pace , and others would hasten after them , and then Spendius would begin again though still torn with agony . This subsided at last on the evening of the fourth day . They were marching side by side to the right of the army on the side of a hill ; below them 
 stretched the plain lost in the vapours of the night . The lines of soldiers also were defiling below , making undulations in the shade . From time to time these passed over eminences lit up by the moon ; then stars would tremble on the points of the pikes , the helmets would glimmer for an instant , all would disappear , and others would come on continually . Startled flocks bleated in the distance , and a something of infinite sweetness seemed to sink upon the earth . Spendius , with his head thrown back and his eyes half-closed , inhaled the freshness of the wind with great sighs ; he spread out his arms , moving his fingers that he might the better feel the cares that streamed over his body . Hopes of vengeance came back to him and transported him . He pressed his hand upon his mouth to check his sobs , and half-swooning with intoxication , let go the halter of his dromedary , which was proceeding with long , regular steps . Matho had relapsed into his former melancholy ; his le
 gs hung down to the ground , and the grass made a continuous rustling as it beat against his cothurni . The journey , however , spread itself out without ever coming to an end . At the extremity of a plain they would always reach a round-shaped plateau ; then they would descend again into a valley , and the mountains which seemed to block up the horizon would , in proportion as they were approached , glide as it were from their positions . From time to time a river would appear amid the verdure of tamarisks to lose itself at the turning of the hills . Sometimes a huge rock would tower aloft like the prow of a vessel or the pedestal of some vanished colossus . At regular intervals they met with little quadrangular temples , which served as stations for the pilgrims who repaired to Sicca . They were closed like tombs . The Libyans struck great blows upon the doors to have them opened . But no one inside responded . Then the cultivation became more rare . They suddenly entered upon bel
 ts of sand bristling with thorny thickets . Flocks of sheep were browsing among the stones ; a woman with a blue fleece about her waist was watching them . She fled screaming when she saw the soldiers ' pikes among the rocks . They were marching through a kind of large passage bordered by two chains of reddish coloured hillocks , when their nostrils were greeted with a nauseous odour , and they thought that they could see something extraordinary on the top of a carob tree : a lion 's head reared itself above the leaves . They ran thither . It was a lion with his four limbs fastened to a cross like a criminal . His huge muzzle fell upon his breast , and his two fore-paws , half-hidden beneath the abundance of his mane , were spread out wide like the wings of a bird . His ribs stood severally out beneath his distended skin ; his hind legs , which were nailed against each other , were raised somewhat , and the black blood , flowing through his hair , had collected in stalactites at the
  end of his tail , which hung down perfectly straight along the cross . The soldiers made merry around ; they called him consul , and Roman citizen , and threw pebbles into his eyes to drive away the gnats . But a hundred paces further on they saw two more , and then there suddenly appeared a long file of crosses bearing lions . Some had been so long dead that nothing was left against the wood but the remains of their skeletons ; others which were half eaten away had their jaws twisted into horrible grimaces ; there were some enormous ones ; the shafts of the crosses bent beneath them , and they swayed in the wind , while bands of crows wheeled ceaselessly in the air above their heads . It was thus that the Carthaginian peasants avenged themselves when they captured a wild beast ; they hoped to terrify the others by such an example . The Barbarians ceased their laughter , and were long lost in amazement . " What people is this , " they thought , " that amuses itself by crucifying li
 ons ! " They were , besides , especially the men of the North , vaguely uneasy , troubled , and already sick . They tore their hands with the darts of the aloes ; great mosquitoes buzzed in their ears , and dysentry was breaking out in the army . They were weary at not yet seeing Sicca . They were afraid of losing themselves and of reaching the desert , the country of sands and terrors . Many even were unwilling to advance further . Others started back to Carthage . At last on the seventh day , after following the base of a mountain for a long time , they turned abruptly to the right , and there then appeared a line of walls resting on white rocks and blending with them . Suddenly the entire city rose ; blue , yellow , and white veils moved on the walls in the redness of the evening . These were the priestesses of Tanith , who had hastened hither to receive the men . They stood ranged along the rampart , striking tabourines , playing lyres , and shaking crotala , while the rays of t
 he sun , setting behind them in the mountains of Numidia , shot between the strings of their lyres over which their naked arms were stretched . At intervals their instruments would become suddenly still , and a cry would break forth strident , precipitate , frenzied , continuous , a sort of barking which they made by striking both corners of the mouth with the tongue . Others , more motionless than the Sphynx , rested on their elbows with their chins on their hands , and darted their great black eyes upon the army as it ascended . Although Sicca was a sacred town it could not hold such a multitude ; the temple alone , with its appurtenances , occupied half of it . Accordingly the Barbarians established themselves at their ease on the plain ; those who were disciplined in regular troops , and the rest according to nationality or their own fancy . The Greeks ranged their tents of skin in parallel lines ; the Iberians placed their canvas pavilions in a circle ; the Gauls made themselve
 s huts of planks ; the Libyans cabins of dry stones , while the Negroes with their nails hollowed out trenches in the sand to sleep in . Many , not knowing where to go , wandered about among the baggage , and at nightfall lay down in their ragged mantles on the ground . The plain , which was wholly bounded by mountains , expanded around them . Here and there a palm tree leaned over a sand hill , and pines and oaks flecked the sides of the precipices : sometimes the rain of a storm would hang from the sky like a long scarf , while the country everywhere was still covered with azure and serenity ; then a warm wind would drive before it tornadoes of dust , and a stream would descend in cascades from the heights of Sicca , where , with its roofing of gold on its columns of brass , rose the temple of the Carthaginian Venus , the mistress of the land . She seemed to fill it with her soul . In such convulsions of the soil , such alternations of temperature , and such plays of light would s
 he manifest the extravagance of her might with the beauty of her eternal smile . The mountains at their summits were crescent-shaped ; others were like women 's bosoms presenting their swelling breasts , and the Barbarians felt a heaviness that was full of delight weighing down their fatigues . Spendius had bought a slave with the money brought him by his dromedary . The whole day long he lay asleep stretched before Matho 's tent . Often he would awake , thinking in his dreams that he heard the whistling of the thongs ; with a smile he would pass his hands over the scars on his legs at the place where the fetters had long been worn , and then he would fall asleep again . Matho accepted his companionship , and when he went out Spendius would escort him like a lictor with a long sword on his thigh ; or perhaps Matho would rest his arm carelessly on the other 's shoulder , for Spendius was small . One evening when they were passing together through the streets in the camp they perceive
 d some men covered with white cloaks ; among them was Narr ' Havas , the prince of the Numidians . Matho started . " Your sword ! " he cried ; " I will kill him ! " " Not yet ! " said Spendius , restraining him . Narr ' Havas was already advancing towards him . He kissed both thumbs in token of alliance , showing nothing of the anger which he had experienced at the drunkenness of the feast ; then he spoke at length against Carthage , but did not say what brought him among the Barbarians . " Was it to betray them , or else the Republic ? " Spendius asked himself ; and as he expected to profit by every disorder , he felt grateful to Narr ' Havas for the future perfidies of which he suspected him . The chief of the Numidians remained amongst the Mercenaries . He appeared desirous of attaching Matho to himself . He sent him fat goats , gold dust , and ostrich feathers . The Libyan , who was amazed at such caresses , was in doubt whether to respond to them or to become exasperated at the
 m . But Spendius pacified him , and Matho allowed himself to be ruled by the slave , remaining ever irresolute and in an unconquerable torpor , like those who have once taken a draught of which they are to die . One morning when all three went out lion-hunting , Narr ' Havas concealed a dagger in his cloak . Spendius kept continually behind him , and when they returned the dagger had not been drawn . Another time Narr ' Havas took them a long way off , as far as the boundaries of his kingdom . They came to a narrow gorge , and Narr ' Havas smiled as he declared that he had forgotten the way . Spendius found it again . But most frequently Matho would go off at sunrise , as melancholy as an augur , to wander about the country . He would stretch himself on the sand , and remain there motionless until the evening . He consulted all the soothsayers in the army one after the other , --those who watch the trail of serpents , those who read the stars , and those who breathe upon the ashes o
 f the dead . He swallowed galbanum , seseli , and viper 's venom which freezes the heart ; Negro women , singing barbarous words in the moonlight , pricked the skin of his forehead with golden stylets ; he loaded himself with necklaces and charms ; he invoked in turn Baal-Khamon , Moloch , the seven Kabiri , Tanith , and the Venus of the Greeks . He engraved a name upon a copper plate , and buried it in the sand at the threshold of his tent . Spendius used to hear him groaning and talking to himself . One night he went in . Matho , as naked as a corpse , was lying on a lion 's skin flat on his stomach , with his face in both his hands ; a hanging lamp lit up his armour , which was hooked on to the tent-pole above his head . " You are suffering ? " said the slave to him . " What is the matter with you ? Answer me ? " And he shook him by the shoulder calling him several times , " Master ! master ! " At last Matho lifted large troubled eyes towards him . " Listen ! " he said in a low v
 oice , and with a finger on his lips . " It is the wrath of the Gods ! Hamilcar 's daughter pursues me ! I am afraid of her , Spendius ! " He pressed himself close against his breast like a child terrified by a phantom . " Speak to me ! I am sick ! I want to get well ! I have tried everything ! But you , you perhaps know some stronger gods , or some resistless invocation ? " " For what purpose ? " asked Spendius . Striking his head with both his fists , he replied : " To rid me of her ! " Then speaking to himself with long pauses he said : " I am no doubt the victim of some holocaust which she has promised to the gods ? --She holds me fast by a chain which people cannot see . If I walk , it is she that is advancing ; when I stop , she is resting ! Her eyes burn me , I hear her voice . She encompasses me , she penetrates me . It seems to me that she has become my soul ! " And yet between us there are , as it were , the invisible billows of a boundless ocean ! She is far away and quit
 e inaccessible ! The splendour of her beauty forms a cloud of light around her , and at times I think that I have never seen her--that she does not exist--and that it is all a dream ! " Matho wept thus in the darkness ; the Barbarians were sleeping . Spendius , as he looked at him , recalled the young men who once used to entreat him with golden cases in their hands , when he led his herd of courtesans through the towns ; a feeling of pity moved him , and he said-- " Be strong , my master ! Summon your will , and beseech the gods no more , for they turn not aside at the cries of men ! Weeping like a coward ! And you are not humiliated that a woman can cause you so much suffering ? " " Am I a child ? " said Matho . " Do you think that I am moved by their faces and songs ? We kept them at Drepanum to sweep out our stables . I have embraced them amid assaults , beneath falling ceilings , and while the catapult was still vibrating ! --But she , Spendius , she ! -- " The slave interrupte
 d him : " If she were not Hanno 's daughter-- " " No ! " cried Matho . " She has nothing in common with the daughters of other men ! Have you seen her great eyes beneath her great eyebrows , like suns beneath triumphal arches ? Think : when she appeared all the torches grew pale . Her naked breast shone here and there through the diamonds of her necklace ; behind her you perceived as it were the odour of a temple , and her whole being emitted something that was sweeter than wine and more terrible than death . She walked , however , and then she stopped . " He remained gaping with his head cast down and his eyeballs fixed . " But I want her ! I need her ! I am dying for her ! I am transported with frenzied joy at the thought of clasping her in my arms , and yet I hate her , Spendius ! I should like to beat her ! What is to be done ? I have a mind to sell myself and become her slave ! YOU have been that ! You were able to get sight of her ; speak to me of her ! Every night she ascends
  to the terrace of her palace , does she not ? Ah ! the stones must quiver beneath her sandals , and the stars bend down to see her ! " He fell back in a perfect frenzy , with a rattling in his throat like a wounded bull . Then Matho sang : " He pursued into the forest the female monster , whose tail undulated over the dead leaves like a silver brook . " And with lingering tones he imitated Salammbo 's voice , while his outspread hands were held like two light hands on the strings of a lyre . To all the consolations offered by Spendius , he repeated the same words ; their nights were spent in these wailings and exhortations . Matho sought to drown his thoughts in wine . After his fits of drunkenness he was more melancholy still . He tried to divert himself at huckle-bones , and lost the gold plates of his necklace one by one . He had himself taken to the servants of the Goddess ; but he came down the hill sobbing , like one returning from a funeral . Spendius , on the contrary , bec
 ame more bold and gay . He was to be seen in the leafy taverns discoursing in the midst of the soldiers . He mended old cuirasses . He juggled with daggers . He went and gathered herbs in the fields for the sick . He was facetious , dexterous , full of invention and talk ; the Barbarians grew accustomed to his services , and he came to be loved by them . However , they were awaiting an ambassador from Carthage to bring them mules laden with baskets of gold ; and ever beginning the same calculation over again , they would trace figures with their fingers in the sand . Every one was arranging his life beforehand ; they would have concubines , slaves , lands ; others intended to bury their treasure , or risk it on a vessel . But their tempers were provoked by want of employment ; there were constant disputes between horse-soldiers and foot-soldiers , Barbarians and Greeks , while there was a never-ending din of shrill female voices . Every day men came flocking in nearly naked , and wi
 th grass on their heads to protect them from the sun ; they were the debtors of the rich Carthaginians and had been forced to till the lands of the latter , but had escaped . Libyans came pouring in with peasants ruined by the taxes , outlaws , and malefactors . Then the horde of traders , all the dealers in wine and oil , who were furious at not being paid , laid the blame upon the Republic . Spendius declaimed against it . Soon the provisions ran low ; and there was talk of advancing in a body upon Carthage , and calling in the Romans . One evening , at supper-time , dull cracked sounds were heard approaching , and something red appeared in the distance among the undulations of the soil . It was a large purple litter , adorned with ostrich feathers at the corners . Chains of crystal and garlands of pearls beat against the closed hangings . It was followed by camels sounding the great bells that hung at their breasts , and having around them horsemen clad from shoulder to heel in a
 rmour of golden scales . They halted three hundred paces from the camp to take their round bucklers , broad swords , and Boeotian helmets out of the cases which they carried behind their saddles . Some remained with the camels , while the others resumed their march . At last the ensigns of the Republic appeared , that is to say , staves of blue wood terminated in horses ' heads or fir cones . The Barbarians all rose with applause ; the women rushed towards the guards of the Legion and kissed their feet . The litter advanced on the shoulders of twelve Negroes who walked in step with short , rapid strides ; they went at random to right or left , being embarrassed by the tent-ropes , the animals that were straying about , or the tripods where food was being cooked . Sometimes a fat hand , laden with rings , would partially open the litter , and a hoarse voice would utter loud reproaches ; then the bearers would stop and take a different direction through the camp . But the purple curta
 ins were raised , and a human head , impassible and bloated , was seen resting on a large pillow ; the eyebrows , which were like arches of ebony , met each other at the points ; golden dust sparkled in the frizzled hair , and the face was so wan that it looked as if it had been powdered with marble raspings . The rest of the body was concealed beneath the fleeces which filled the litter . In the man so reclining the soldiers recognised the Suffet Hanno , he whose slackness had assisted to lose the battle of the Aegatian islands ; and as to his victory at Hecatompylos over the Libyans , even if he did behave with clemency , thought the Barbarians , it was owing to cupidity , for he had sold all the captives on his own account , although he had reported their deaths to the Republic . After seeking for some time a convenient place from which to harangue the soldiers , he made a sign ; the litter stopped , and Hanno , supported by two slaves , put his tottering feet to the ground . He 
 wore boots of black felt strewn with silver moons . His legs were swathed in bands like those wrapped about a mummy , and the flesh crept through the crossings of the linen ; his stomach came out beyond the scarlet jacket which covered his thighs ; the folds of his neck fell down to his breast like the dewlaps of an ox ; his tunic , which was painted with flowers , was bursting at the arm-pits ; he wore a scarf , a girdle , and an ample black cloak with laced double-sleeves . But the abundance of his garments , his great necklace of blue stones , his golden clasps , and heavy earrings only rendered his deformity still more hideous . He might have been taken for some big idol rough-hewn in a block of stone ; for a pale leprosy , which was spread over his whole body , gave him the appearance of an inert thing . His nose , however , which was hooked like a vulture 's beak , was violently dilated to breathe in the air , and his little eyes , with their gummed lashes , shone with a hard 
 and metallic lustre . He held a spatula of aloe-wood in his hand wherewith to scratch his skin . At last two heralds sounded their silver horns ; the tumult subsided , and Hanno commenced to speak . He began with an eulogy of the gods and the Republic ; the Barbarians ought to congratulate themselves on having served it . But they must show themselves more reasonable ; times were hard , " and if a master has only three olives , is it not right that he should keep two for himself ? " The old Suffet mingled his speech in this way with proverbs and apologues , nodding his head the while to solicit some approval . He spoke in Punic , and those surrounding him ( the most alert , who had hastened thither without their arms ) , were Campanians , Gauls , and Greeks , so that no one in the crowd understood him . Hanno , perceiving this , stopped and reflected , swaying himself heavily from one leg to the other . It occurred to him to call the captains together ; then his heralds shouted the 
 order in Greek , the language which , from the time of Xanthippus , had been used for commands in the Carthaginian armies . The guards dispersed the mob of soldiers with strokes of the whip ; and the captains of the Spartan phalanxes and the chiefs of the Barbarian cohorts soon arrived with the insignia of their rank , and in the armour of their nation . Night had fallen , a great tumult was spreading throughout the plain ; fires were burning here and there ; and the soldiers kept going from one to another asking what the matter was , and why the Suffet did not distribute the money ? He was setting the infinite burdens of the Republic before the captains . Her treasury was empty . The tribute to Rome was crushing her . " We are quite at a loss what to do ! She is much to be pitied ! " From time to time he would rub his limbs with his aloe-wood spatula , or perhaps he would break off to drink a ptisan made of the ashes of a weasel and asparagus boiled in vinegar from a silver cup han
 ded to him by a slave ; then he would wipe his lips with a scarlet napkin and resume : " What used to be worth a shekel of silver is now worth three shekels of gold , while the cultivated lands which were abandoned during the war bring in nothing ! Our purpura fisheries are nearly gone , and even pearls are becoming exhorbitant ; we have scarcely unguents enough for the service of the gods ! As for the things of the table , I shall say nothing about them ; it is a calamity ! For want of galleys we are without spices , and it is a matter of great difficulty to procure silphium on account of the rebellions on the Cyrenian frontier . Sicily , where so many slaves used to be had , is now closed to us ! Only yesterday I gave more money for a bather and four scullions than I used at one time to give for a pair of elephants ! " He unrolled a long piece of papyrus ; and , without omitting a single figure , read all the expenses that the government had incurred ; so much for repairing the te
 mples , for paving the streets , for the construction of vessels , for the coral-fisheries , for the enlargement of the Syssitia , and for engines in the mines in the country of the Cantabrians . But the captains understood Punic as little as the soldiers , although the Mercenaries saluted one another in that language . It was usual to place a few Carthaginian officers in the Barbarian armies to act as interpreters ; after the war they had concealed themselves through fear of vengeance , and Hanno had not thought of taking them with him ; his hollow voice , too , was lost in the wind . The Greeks , girthed in their iron waist-belts , strained their ears as they strove to guess at his words , while the mountaineers , covered with furs like bears , looked at him with distrust , or yawned as they leaned on their brass-nailed clubs . The heedless Gauls sneered as they shook their lofty heads of hair , and the men of the desert listened motionless , cowled in their garments of grey wool 
 ; others kept coming up behind ; the guards , crushed by the mob , staggered on their horses ; the Negroes held out burning fir branches at arm 's length ; and the big Carthaginian , mounted on a grassy hillock , continued his harangue . The Barbarians , however , were growing impatient ; murmuring arose , and every one apostrophized him . Hanno gesticulated with his spatula ; and those who wished the others to be quiet shouted still more loudly , thereby adding to the din . Suddenly a man of mean appearance bounded to Hanno 's feet , snatched up a herald 's trumpet , blew it , and Spendius ( for it was he ) announced that he was going to say something of importance . At this declaration , which was rapidly uttered in five different languages , Greek , Latin , Gallic , Libyan and Balearic , the captains , half laughing and half surprised , replied : " Speak ! Speak ! " Spendius hesitated ; he trembled ; at last , addressing the Libyans who were the most numerous , he said to them : 
 " You have all heard this man 's horrible threats ! " Hanno made no exclamation , therefore he did not understand Libyan ; and , to carry on the experiment , Spendius repeated the same phrase in the other Barbarian dialects . They looked at one another in astonishment ; then , as by a tacit agreement , and believing perhaps that they had understood , they bent their heads in token of assent . Then Spendius began in vehement tones : " He said first that all the Gods of the other nations were but dreams besides the Gods of Carthage ! He called you cowards , thieves , liars , dogs , and the sons of dogs ! But for you ( he said that ! ) the Republic would not be forced to pay excessive tribute to the Romans ; and through your excesses you have drained it of perfumes , aromatics , slaves , and silphium , for you are in league with the nomads on the Cyrenian frontier ! But the guilty shall be punished ! He read the enumeration of their torments ; they shall be made to work at the paving o
 f the streets , at the equipment of the vessels , at the adornment of the Syssitia , while the rest shall be sent to scrape the earth in the mines in the country of the Cantabrians . " Spendius repeated the same statements to the Gauls , Greeks , Campanians and Balearians . The Mercenaries , recognising several of the proper names which had met their ears , were convinced that he was accurately reporting the Suffet 's speech . A few cried out to him , " You lie ! " but their voices were drowned in the tumult of the rest ; Spendius added : " Have you not seen that he has left a reserve of his horse-soldiers outside the camp ? At a given signal they will hasten hither to slay you all . " The Barbarians turned in that direction , and as the crowd was then scattering , there appeared in the midst of them , and advancing with the slowness of a phantom , a human being , bent , lean , entirely naked , and covered down to his flanks with long hair bristling with dried leaves , dust and thor
 ns . About his loins and his knees he had wisps of straw and linen rags ; his soft and earthy skin hung on his emaciated limbs like tatters on dried boughs ; his hands trembled with a continuous quivering , and as he walked he leaned on a staff of olive-wood . He reached the Negroes who were bearing the torches . His pale gums were displayed in a sort of idiotic titter ; his large , scared eyes gazed upon the crowd of Barbarians around him . But uttering a cry of terror he threw himself behind them , shielding himself with their bodies . " There they are ! There they are ! " he stammered out , pointing to the Suffet 's guards , who were motionless in their glittering armour . Their horses , dazzled by the light of the torches which crackled in the darkness , were pawing the ground ; the human spectre struggled and howled : " They have killed them ! " At these words , which were screamed in Balearic , some Balearians came up and recognised him ; without answering them he repeated : "
  Yes , all killed , all ! crushed like grapes ! The fine young men ! the slingers ! my companions and yours ! " They gave him wine to drink , and he wept ; then he launched forth into speech . Spendius could scarcely repress his joy , as he explained the horrors related by Zarxas to the Greeks and Libyans ; he could not believe them , so appropriately did they come in . The Balearians grew pale as they learned how their companions had perished . It was a troop of three hundred slingers who had disembarked the evening before , and had on that day slept too late . When they reached the square of Khamon the Barbarians were gone , and they found themselves defenceless , their clay bullets having been put on the camels with the rest of the baggage . They were allowed to advance into the street of Satheb as far as the brass sheathed oaken gate ; then the people with a single impulse had sprung upon them . Indeed , the soldiers remembered a great shout ; Spendius , who was flying at the he
 ad of the columns , had not heard it . Then the corpses were placed in the arms of the Pataec gods that fringed the temple of Khamon . They were upbraided with all the crimes of the Mercenaries ; their gluttony , their thefts , their impiety , their disdain , and the murder of the fishes in Salammbo 's garden . Their bodies were subjected to infamous mutilations ; the priests burned their hair in order to torture their souls ; they were hung up in pieces in the meat-shops ; some even buried their teeth in them , and in the evening funeral-piles were kindled at the cross-ways to finish them . These were the flames that had gleamed from a distance across the lake . But some houses having taken fire , any dead or dying that remained were speedily thrown over the walls ; Zarxas had remained among the reeds on the edge of the lake until the following day ; then he had wandered about through the country , seeking for the army by the footprints in the dust . In the morning he hid himself i
 n caves ; in the evening he resumed his march with his bleeding wounds , famished , sick , living on roots and carrion ; at last one day he perceived lances on the horizon , and he had followed them , for his reason was disturbed through his terrors and miseries . The indignation of the soldiers , restrained so long as he was speaking , broke forth like a tempest ; they were going to massacre the guards together with the Suffet . A few interposed , saying that they ought to hear him and know at least whether they should be paid . Then they all cried : " Our money ! " Hanno replied that he had brought it . They ran to the outposts , and the Suffet 's baggage arrived in the midst of the tents , pressed forward by the Barbarians . Without waiting for the slaves , they very quickly unfastened the baskets ; in them they found hyacinth robes , sponges , scrapers , brushes , perfumes , and antimony pencils for painting the eyes--all belonging to the guards , who were rich men and accustome
 d to such refinements . Next they uncovered a large bronze tub on a camel : it belonged to the Suffet who had it for bathing in during his journey ; for he had taken all manner of precautions , even going so far as to bring caged weasels from Hecatompylos , which were burnt alive to make his ptisan . But , as his malady gave him a great appetite , there were also many comestibles and many wines , pickle , meats and fishes preserved in honey , with little pots of Commagene , or melted goose-fat covered with snow and chopped straw . There was a considerable supply of it ; the more they opened the baskets the more they found , and laughter arose like conflicting waves . As to the pay of the Mercenaries it nearly filled two esparto-grass baskets ; there were even visible in one of them some of the leathern discs which the Republic used to economise its specie ; and as the Barbarians appeared greatly surprised , Hanno told them that , their accounts being very difficult , the Ancients ha
 d not had leisure to examine them . Meanwhile they had sent them this . Then everything was in disorder and confusion : mules , serving men , litter , provisions , and baggage . The soldiers took the coin in the bags to stone Hanno . With great difficulty he was able to mount an ass ; and he fled , clinging to its hair , howling , weeping , shaken , bruised , and calling down the curse of all the gods upon the army . His broad necklace of precious stones rebounded up to his ears . His cloak which was too long , and which trailed behind him , he kept on with his teeth , and from afar the Barbarians shouted at him , " Begone coward ! pig ! sink of Moloch ! sweat your gold and your plague ! quicker ! quicker ! " The routed escort galloped beside him . But the fury of the Barbarians did not abate . They remembered that several of them who had set out for Carthage had not returned ; no doubt they had been killed . So much injustice exasperated them , and they began to pull up the stakes 
 of their tents , to roll up their cloaks , and to bridle their horses ; every one took his helmet and sword , and instantly all was ready . Those who had no arms rushed into the woods to cut staves . Day dawned ; the people of Sicca were roused , and stirring in the streets . " They are going to Carthage , " said they , and the rumour of this soon spread through the country . From every path and every ravine men arose . Shepherds were seen running down from the mountains . Then , when the Barbarians had set out , Spendius circled the plain , riding on a Punic stallion , and attended by his slave , who led a third horse . A single tent remained . Spendius entered it . " Up , master ! rise ! we are departing ! " " And where are you going ? " asked Matho . " To Carthage ! " cried Spendius . Matho bounded upon the horse which the slave held at the door . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/60FictGrimmJ_Bremusicians_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/60FictGrimmJ_Bremusicians_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/60FictGrimmJ_Bremusicians_EN.txt.txt
new file mode 100644
index 0000000..7f29fa0
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/60FictGrimmJ_Bremusicians_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff The Bremen town musicians There was once an ass whose master had made him carry sacks to the mill for many a long year , but whose strength began at last to fail , so that each day as it came found him less capable of work . Then his master began to think of turning him out , but the ass , guessing that something was in the wind that boded him no good , ran away , taking the road to Bremen ; for there he thought he might get an engagement as town musician . When he had gone a little way he found a hound lying by the side of the road panting , as if he had run a long way . \u201c Now , Holdfast , what are you so out of breath about ? \u201d said the ass . \u201c Oh dear ! \u201d said the dog , \u201c now I am old , I get weaker every day , and can do no good in the hunt , so , as my master was going to have me killed , I have made my escape ; but now , how am I to gain a living ? \u201d - \u201c I will tell you what , \u201d said the ass , \u201c I am going to Bremen to become town musician . You may as we
 ll go with me , and take up music too . I can play the lute , and you can beat the drum . \u201d And the dog consented , and they walked on together . It was not long before they came to a cat sitting in the road , looking as dismal as three wet days. \u201c Now then , what is the matter with you , old shaver ? \u201d said the ass . \u201c I should like to know who would be cheerful when his neck is in danger , \u201d answered the cat . \u201c Now that I am old my teeth are getting blunt , and I would rather sit by the oven and purr than run about after mice , and my mistress wanted to drown me ; so I took myself off ; but good advice is scarce , and I do not know what is to become of me . \u201d - \u201c Go with us to Bremen , \u201d said the ass , \u201c and become town musician . You understand serenading . \u201d The cat thought well of the idea , and went with them accordingly . After that the three travellers passed by a yard , and a cock was perched on the gate crowing with all his might . \u201c Your cries are
  enough to pierce bone and marrow , \u201d said the ass ; \u201c what is the matter ? \u201d - \u201c I have foretold good weather for Lady-day , so that all the shirts may be washed and dried ; and now on Sunday morning company is coming , and the mistress has told the cook that I must be made into soup , and this evening my neck is to be wrung , so that I am crowing with all my might while I can . \u201d - \u201c You had much better go with us , Chanticleer , \u201d said the ass . \u201c We are going to Bremen . At any rate that will be better than dying . You have a powerful voice , and when we are all performing together it will have a very good effect . \u201d So the cock consented , and they went on all four together . But Bremen was too far off to be reached in one day , and towards evening they came to a wood , where they determined to pass the night . The ass and the dog lay down under a large tree ; the cat got up among the branches , and the cock flew up to the top , as that was the safest place fo
 r him . Before he went to sleep he looked all round him to the four points of the compass , and perceived in the distance a little light shining , and he called out to his companions that there must be a house not far off , as he could see a light , so the ass said , \u201c We had better get up and go there , for these are uncomfortable quarters . \u201d The dog began to fancy a few bones , not quite bare , would do him good . And they all set off in the direction of the light , and it grew larger and brighter , until at last it led them to a robber\u2019s house , all lighted up . The ass . being the biggest , went up to the window , and looked in . \u201c Well , what do you see ? \u201d asked the dog . \u201c What do I see ? \u201d answered the ass ; \u201c here is a table set out with splendid eatables and drinkables , and robbers sitting at it and making themselves very comfortable . \u201d - \u201c That would just suit us , \u201d said the cock . \u201c Yes , indeed , I wish we were there , \u201d said the ass . Then
  they consulted together how it should be managed so as to get the robbers out of the house , and at last they hit on a plan . The ass was to place his forefeet on the window-sill , the dog was to get on the ass\u2019s back , the cat on the top of the dog , and lastly the cock was to fly up and perch on the cat\u2019s head . When that was done , at a given signal they all began to perform their music . The ass brayed , the dog barked , the cat mewed , and the cock crowed ; then they burst through into the room , breaking all the panes of glass . The robbers fled at the dreadful sound ; they thought it was some goblin , and fled to the wood in the utmost terror . Then the four companions sat down to table , made free with the remains of the meal , and feasted as if they had been hungry for a month . And when they had finished they put out the lights , and each sought out a sleeping-place to suit his nature and habits . The ass laid himself down outside on the dunghill , the dog behind the 
 door , the cat on the hearth by the warm ashes , and the cock settled himself in the cockloft , and as they were all tired with their long journey they soon fell fast asleep . When midnight drew near , and the robbers from afar saw that no light was burning , and that everything appeared quiet , their captain said to them that he thought that they had run away without reason , telling one of them to go and reconnoitre . So one of them went , and found everything quite quiet ; he went into the kitchen to strike a light , and taking the glowing fiery eyes of the cat for burning coals , he held a match to them in order to kindle it . But the cat , not seeing the joke , flew into his face , spitting and scratching . Then he cried out in terror , and ran to get out at the back door , but the dog , who was lying there , ran at him and bit his leg ; and as he was rushing through the yard by the dunghill the ass struck out and gave him a great kick with his hind foot ; and the cock , who ha
 d been wakened with the noise , and felt quite brisk , cried out , \u201c Cock-a-doodle-doo ! \u201d Then the robber got back as well as he could to his captain , and said , \u201c Oh dear ! in that house there is a gruesome witch , and I felt her breath and her long nails in my face ; and by the door there stands a man who stabbed me in the leg with a knife ; and in the yard there lies a black spectre , who beat me with his wooden club ; and above , upon the roof , there sits the justice , who cried , \u2018 Bring that rogue here ! \u2019 And so I ran away from the place as fast as I could . \u201d From that time forward the robbers never ventured to that house , and the four Bremen town musicians found themselves so well off where they were , that there they stayed . And the person who last related this tale is still living , as you see . 
\ No newline at end of file


[45/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/NamedEntityExtractor.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/NamedEntityExtractor.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/NamedEntityExtractor.java
new file mode 100644
index 0000000..b766c7c
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/NamedEntityExtractor.java
@@ -0,0 +1,185 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.parse_thicket.opinion_processor;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import opennlp.tools.apps.relevanceVocabs.SentimentVocab;
+import opennlp.tools.jsmlearning.ProfileReaderWriter;
+import opennlp.tools.parse_thicket.ParseThicket;
+import opennlp.tools.parse_thicket.ParseTreeNode;
+import opennlp.tools.parse_thicket.VerbNetProcessor;
+import opennlp.tools.parse_thicket.kernel_interface.DescriptiveParagraphFromDocExtractor;
+import opennlp.tools.parse_thicket.matching.Matcher;
+import opennlp.tools.parse_thicket.matching.PT2ThicketPhraseBuilder;
+import opennlp.tools.similarity.apps.utils.Pair;
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+import opennlp.tools.textsimilarity.TextProcessor;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang.StringUtils;
+
+public class NamedEntityExtractor {
+	protected static Matcher matcher;
+	private static int PARA_LENGTH_IN_SENTENCES = 5, PARA_LENGTH = 250;
+	protected ArrayList<File> queue = new ArrayList<File>();
+	protected static PT2ThicketPhraseBuilder phraseBuilder;
+	protected static SentimentVocab sVocab = SentimentVocab.getInstance();
+	String resourceDirSentimentList = null;
+	Set<String> sentimentVcb = new HashSet<String> ();
+
+	static {
+		synchronized (NamedEntityExtractor.class) {
+			matcher = new Matcher();
+			phraseBuilder = new PT2ThicketPhraseBuilder();
+		}
+	}
+
+	public NamedEntityExtractor(){
+		try {
+			resourceDirSentimentList = new File( "." ).getCanonicalPath()+"/src/test/resources/opinions/sentiment_listReduced.csv";
+		} catch (IOException e) {
+			e.printStackTrace();
+		}
+		List<String[]> sentimentList=null;
+		sentimentList = ProfileReaderWriter.readProfiles(resourceDirSentimentList);
+		for(String[] line: sentimentList){
+			sentimentVcb.add(line[0]);
+		}
+	}
+
+	protected boolean isSentimentWord(String word){
+		if (sentimentVcb.contains(word))
+			return true;
+		else
+			return false;		
+	}
+
+	public EntityExtractionResult extractEntities(String para){
+		List<List<ParseTreeNode>> extractedNERs = new ArrayList<List<ParseTreeNode>>();
+		List<String> extractedNERsWords = new ArrayList<String>();
+		List<List<ParseTreeNode>> extractedSentimentPhrases = 
+				new ArrayList<List<ParseTreeNode>>();
+		EntityExtractionResult result = new EntityExtractionResult();
+
+		ParseThicket pt = null;
+
+		System.out.println("Processing paragraph of length "+para.length() + " | "+ para);
+		pt = matcher.buildParseThicketFromTextWithRST(para);
+		List<List<ParseTreeNode>> nodeList = pt.getSentenceNodes();
+
+
+		for(List<ParseTreeNode> sentence: nodeList){
+			//System.out.println("   Processing sentence: "+ sentence);
+			boolean bInsideNER = false; 
+			String currentPhrase = "";
+			List<ParseTreeNode> currentPhraseNode = new ArrayList<ParseTreeNode>(); 
+			for(ParseTreeNode word: sentence){
+				if (isNERforPhraseExtraction(word)){
+					//System.out.println("++Found word ="+word + " | NER="+ word.getNe());
+					if (bInsideNER){
+						currentPhrase += " "+word.getWord();
+						currentPhraseNode.add(word);
+					} else {
+						bInsideNER=true;
+						currentPhrase = word.getWord();
+						currentPhraseNode.add(word);
+					}
+				} else {
+					if (bInsideNER){
+						if (currentPhrase.indexOf(' ')>-1) // at least two tokens
+							extractedNERsWords.add(currentPhrase);
+							extractedNERs.add(currentPhraseNode);
+						currentPhrase = "";
+						bInsideNER=false;
+					} else {
+						// do nothing, continue scan
+					}
+				}
+			}
+			if (currentPhrase.length()>1 && currentPhrase.indexOf(' ')>-1){
+				extractedNERs.add(currentPhraseNode);
+				extractedNERsWords.add(currentPhrase);
+			}
+
+			Set<String> foundSentimentWords = new HashSet<String>();
+			// now we extract phrases
+			List<List<ParseTreeNode>> phrases = pt.getPhrases();
+			for(List<ParseTreeNode> phrase: phrases){
+				// find a noun phrase under sentiment
+				try {
+					for(int i = phrase.size()-1; i>-1; i--){
+						ParseTreeNode word = phrase.get(i);
+						if ((isSentimentWord(word.getWord()) ||
+								sVocab.isSentimentWord(word.getWord()) && !foundSentimentWords.contains(word.getWord()) )){
+							foundSentimentWords.add(word.getWord());
+							System.out.println("Sentim = " + word.getWord() + " | Found opinionated phrase "+phrase.toString());
+							if (phrase.size()>1 && phrase.size()<7)
+								extractedSentimentPhrases.add(phrase);			
+							break;
+						}
+					}
+				} catch (Exception e) {
+					e.printStackTrace();
+				}
+			}
+
+		} 
+		
+		extractedSentimentPhrases = reduceExtractedPhrases(extractedSentimentPhrases);
+		
+		result.setExtractedNER(extractedNERs);
+		result.setExtractedNERWords(extractedNERsWords);
+		result.setExtractedSentimentPhrases(extractedSentimentPhrases);
+		return result;
+	}
+
+	private List<List<ParseTreeNode>> reduceExtractedPhrases(List<List<ParseTreeNode>> extractedSentimentPhrases) {
+	    List<Integer> idsToDelete = new ArrayList<Integer>();
+		for(int i = 0; i<extractedSentimentPhrases.size(); i++){
+			for(int j = i+1; j<extractedSentimentPhrases.size(); j++){
+				String phrStr1 = ParseTreeNode.toWordString(extractedSentimentPhrases.get(i));
+				String phrStr2 = ParseTreeNode.toWordString(extractedSentimentPhrases.get(j));
+				if (phrStr1 .indexOf(phrStr2 )>-1)
+					idsToDelete.add(j);
+			}
+		}
+		List<List<ParseTreeNode>> resultPhrases = new ArrayList<List<ParseTreeNode>>();
+		for(int i = 0; i<extractedSentimentPhrases.size(); i++){
+			if (!idsToDelete.contains(i))
+				resultPhrases .add(extractedSentimentPhrases.get(i));
+		}
+	    return resultPhrases ;
+    }
+
+	private boolean isNERforPhraseExtraction(ParseTreeNode word){
+		if ((word.getNe().equals("ORGANIZATION") ||word.getNe().equals("LOCATION") || word.getNe().equals("PERSON") ) &&
+				(word.getPos().startsWith("NN") || word.getPos().startsWith("PR") || word.getPos().startsWith("IN")|| 
+						word.getPos().startsWith("JJ") || word.getPos().startsWith("DT")  ))
+			return true;
+
+		return false;
+
+	}
+
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/PersonExtractor.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/PersonExtractor.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/PersonExtractor.java
new file mode 100644
index 0000000..cb04154
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/PersonExtractor.java
@@ -0,0 +1,96 @@
+package opennlp.tools.parse_thicket.opinion_processor;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import opennlp.tools.parse_thicket.ParseThicket;
+import opennlp.tools.parse_thicket.ParseTreeNode;
+
+public class PersonExtractor extends NamedEntityExtractor {
+	private boolean isNERforPhraseExtraction(ParseTreeNode word){
+		if ((word.getNe().equals("PERSON") ) &&
+				(word.getPos().startsWith("NN") || word.getPos().startsWith("PR") || word.getPos().startsWith("IN")|| 
+						word.getPos().startsWith("JJ") || word.getPos().startsWith("DT")  ))
+			return true;
+
+		return false;
+
+	}
+	
+	public EntityExtractionResult extractEntities(String para){
+		List<List<ParseTreeNode>> extractedNERs = new ArrayList<List<ParseTreeNode>>();
+		List<String> extractedNERsWords = new ArrayList<String>();
+		List<List<ParseTreeNode>> extractedSentimentPhrases = 
+				new ArrayList<List<ParseTreeNode>>();
+		EntityExtractionResult result = new EntityExtractionResult();
+
+		ParseThicket pt = null;
+
+		System.out.println("Processing paragraph of length "+para.length() + " | "+ para);
+		pt = matcher.buildParseThicketFromTextWithRST(para);
+		List<List<ParseTreeNode>> nodeList = pt.getSentenceNodes();
+
+
+		for(List<ParseTreeNode> sentence: nodeList){
+			System.out.println("   Processing sentence: "+ sentence);
+			boolean bInsideNER = false; 
+			String currentPhrase = "";
+			List<ParseTreeNode> currentPhraseNode = new ArrayList<ParseTreeNode>(); 
+			for(ParseTreeNode word: sentence){
+				if (isNERforPhraseExtraction(word)){
+					System.out.println("++Found word ="+word + " | NER="+ word.getNe());
+					if (bInsideNER){
+						currentPhrase += " "+word.getWord();
+						currentPhraseNode.add(word);
+					} else {
+						bInsideNER=true;
+						currentPhrase = word.getWord();
+						currentPhraseNode.add(word);
+					}
+				} else {
+					if (bInsideNER){
+						if (currentPhrase.indexOf(' ')>-1) // at least two tokens
+							extractedNERsWords.add(currentPhrase);
+							extractedNERs.add(currentPhraseNode);
+						currentPhrase = "";
+						bInsideNER=false;
+					} else {
+						// do nothing, continue scan
+					}
+				}
+			}
+			if (currentPhrase.length()>1 && currentPhrase.indexOf(' ')>-1){
+				extractedNERs.add(currentPhraseNode);
+				extractedNERsWords.add(currentPhrase);
+			}
+
+			Set<String> foundSentimentWords = new HashSet<String>();
+			// now we extract phrases
+			List<List<ParseTreeNode>> phrases = phraseBuilder.buildPT2ptPhrases(pt);
+			for(List<ParseTreeNode> phrase: phrases){
+				// find a noun phrase under sentiment
+				try {
+					for(int i = phrase.size()-1; i>-1; i--){
+						ParseTreeNode word = phrase.get(i);
+						if ((isSentimentWord(word.getWord()) ||
+								sVocab.isSentimentWord(word.getWord()) && !foundSentimentWords.contains(word.getWord()) )){
+							foundSentimentWords.add(word.getWord());
+							System.out.println("Found opinionated phrase "+phrase.toString());
+							extractedSentimentPhrases.add(phrase);			
+							break;
+						}
+					}
+				} catch (Exception e) {
+					e.printStackTrace();
+				}
+			}
+
+		} 
+		result.setExtractedNER(extractedNERs);
+		result.setExtractedNERWords(extractedNERsWords);
+		result.setExtractedSentimentPhrases(extractedSentimentPhrases);
+		return result;
+	}
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/SentencePhraseGivenAWordGetter.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/SentencePhraseGivenAWordGetter.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/SentencePhraseGivenAWordGetter.java
new file mode 100644
index 0000000..86cd2dc
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/SentencePhraseGivenAWordGetter.java
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.parse_thicket.opinion_processor;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import opennlp.tools.apps.relevanceVocabs.SentimentVocab;
+import opennlp.tools.jsmlearning.ProfileReaderWriter;
+import opennlp.tools.parse_thicket.ParseThicket;
+import opennlp.tools.parse_thicket.ParseTreeNode;
+import opennlp.tools.parse_thicket.VerbNetProcessor;
+import opennlp.tools.parse_thicket.kernel_interface.DescriptiveParagraphFromDocExtractor;
+import opennlp.tools.parse_thicket.matching.Matcher;
+import opennlp.tools.parse_thicket.matching.PT2ThicketPhraseBuilder;
+import opennlp.tools.similarity.apps.utils.Pair;
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+import opennlp.tools.textsimilarity.TextProcessor;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang.StringUtils;
+
+public class SentencePhraseGivenAWordGetter {
+	protected static Matcher matcher;
+	protected ArrayList<File> queue = new ArrayList<File>();
+	protected static PT2ThicketPhraseBuilder phraseBuilder;
+
+
+	static {
+		synchronized (SentencePhraseGivenAWordGetter.class) {
+			matcher = new Matcher();
+			phraseBuilder = new PT2ThicketPhraseBuilder();
+		}
+	}
+
+	public SentencePhraseGivenAWordGetter(){
+	}
+
+	public EntityExtractionResult extractEntities(String para, String keyword){
+		List<List<ParseTreeNode>> extractedPhrases = new ArrayList<List<ParseTreeNode>>();
+
+		EntityExtractionResult result = new EntityExtractionResult();
+
+		ParseThicket pt =  matcher.buildParseThicketFromTextWithRST(para);
+
+		List<List<ParseTreeNode>> phrases = pt.getPhrases();
+		for(List<ParseTreeNode> phrase: phrases){
+			// find a noun phrase under sentiment
+			try {
+				for(int i = 0; i<phrase.size(); i++){
+					ParseTreeNode word = phrase.get(i);
+					if (word.getWord().toLowerCase().equals(keyword.toLowerCase())){
+						extractedPhrases.add(phrase);		
+						break;
+					}
+				}
+			} catch (Exception e) {
+				e.printStackTrace();
+			}
+		}
+
+		result.setExtractedSentimentPhrases(extractedPhrases);
+		return result;
+	}
+
+
+	public static void main(String[] args){
+		SentencePhraseGivenAWordGetter self = new SentencePhraseGivenAWordGetter();
+		EntityExtractionResult result = self.extractEntities("However i put a foam panel inside the main case if i do not have my headphones or an iPad to brace the mac book", 
+				"panel");
+		System.out.println(result.getExtractedSentimentPhrases());
+	}
+}
+
+
+/*
+ 3 phrases are given as a result
+ * 
+[[<2>SBAR'i':FW, <3>SBAR'put':VBD, <4>SBAR'a':DT, <5>SBAR'foam':NN, <6>SBAR'panel':NN, <7>SBAR'inside':IN, <8>SBAR'the':DT, <9>SBAR'main':JJ, <10>SBAR'case':NN, <11>SBAR'if':IN, <12>SBAR'i':FW, 
+<13>SBAR'do':VBP, <14>SBAR'not':RB, <15>SBAR'have':VB, <16>SBAR'my':PRP$, <17>SBAR'headphones':NNS, <18>SBAR'or':CC, <19>SBAR'an':DT, <20>SBAR'iPad':NN, <21>SBAR'to':TO, 
+<22>SBAR'brace':VB, <23>SBAR'the':DT, <24>SBAR'mac':NN, <25>SBAR'book':NN], 
+
+[<3>VP'put':VBD, <4>VP'a':DT, <5>VP'foam':NN, <6>VP'panel':NN, <7>VP'inside':IN, <8>VP'the':DT, <9>VP'main':JJ, <10>VP'case':NN, <11>VP'if':IN, <12>VP'i':FW, <13>VP'do':VBP, 
+<14>VP'not':RB, <15>VP'have':VB, <16>VP'my':PRP$, <17>VP'headphones':NNS, <18>VP'or':CC, <19>VP'an':DT, <20>VP'iPad':NN, <21>VP'to':TO, <22>VP'brace':VB, <23>VP'the':DT, 
+<24>VP'mac':NN, <25>VP'book':NN], 
+
+[<4>NP'a':DT, <5>NP'foam':NN, <6>NP'panel':NN]]
+
+*/

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/SentimentCoreAnnotations.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/SentimentCoreAnnotations.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/SentimentCoreAnnotations.java
new file mode 100644
index 0000000..1efe428
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/SentimentCoreAnnotations.java
@@ -0,0 +1,41 @@
+package opennlp.tools.parse_thicket.opinion_processor;
+
+import edu.stanford.nlp.ling.CoreAnnotation;
+
+import edu.stanford.nlp.trees.Tree;
+
+/**
+ * Annotations specific to the Sentiment project.  In case there are
+ * other projects that use the same RNN machinery, including the RNN
+ * core annotations, this lets a sentence have a tree attached where
+ * that tree specifically has the sentiment annotations.
+ *
+ * @author John Bauer
+ */
+public class SentimentCoreAnnotations {
+
+  /**
+   * A tree which contains the annotations used for the Sentiment
+   * task.  After forwardPropagate has been called, the Tree will have
+   * prediction, etc. attached to it.
+   */
+  public static class SentimentAnnotatedTree implements CoreAnnotation<Tree> {
+    @Override
+    public Class<Tree> getType() {
+      return Tree.class;
+    }
+  }
+
+
+  /**
+   * The final label given for a sentence.  Set by the
+   * SentimentAnnotator and used by various forms of text output.
+   */
+  public static class SentimentClass implements CoreAnnotation<String> {
+    @Override
+    public Class<String> getType() {
+      return String.class;
+    }
+  }
+}
+

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/StopList.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/StopList.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/StopList.java
new file mode 100755
index 0000000..ad0f791
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/StopList.java
@@ -0,0 +1,401 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.parse_thicket.opinion_processor;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Hashtable;
+import java.util.Iterator;
+import java.util.List;
+
+import opennlp.tools.stemmer.PStemmer;
+
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+
+
+public class StopList {
+    private static StopList m_StopList = null;
+    private static Hashtable<String, HashSet<String>> m_stopHash = new Hashtable<String, HashSet<String>>();
+    public static final Log logger = LogFactory.getLog(StopList.class);
+    private static final String DEFAULT_STOPLIST = "STANDARD";
+    public static String resourceDir =null;
+    private static PStemmer stemmer = new PStemmer();
+
+    static {
+        synchronized (StopList.class) {
+            try {
+                LoadStopList();
+            } catch (IOException e) {
+                // TODO Auto-generated catch block
+                e.printStackTrace();
+            }
+        }
+    }
+
+    /**
+     * Get the StopList singleton instance.
+     * 
+     * @return The StopList
+     */
+    static public synchronized StopList getInstance() {
+
+        if (m_StopList == null) {
+            m_StopList = new StopList();
+
+            try {
+                m_StopList.LoadStopList();
+            } catch (Exception e) {
+
+            }
+        }
+        return m_StopList;
+    }
+
+    static public synchronized StopList getInstance(String dir) {
+        resourceDir = dir;
+        if (m_StopList == null) {
+            m_StopList = new StopList();
+
+            try {
+                m_StopList.LoadStopList();
+            } catch (Exception e) {
+
+            }
+        }
+        return m_StopList;
+    }
+
+    private static void LoadStopList() throws IOException {
+
+        File dir = new File(resourceDir + "/maps");
+        String[] children = dir.list();
+        if (children == null) {
+            System.err.println("Problem reading Stop Lists!");
+        } else {
+            for (int i = 0; i < children.length; i++) {
+                String fn = children[i];
+                if (fn.endsWith(".vcb")) {
+                    String fileName = resourceDir + "/maps/" + fn;
+                    File f = new File(fileName);
+                    loadStopListFile(f);
+                }
+            }
+        }
+    }
+
+    private static void loadStopListFile(File f) throws FileNotFoundException {
+
+        FileReader fileReader = new FileReader(f);
+        BufferedReader in = new BufferedReader(fileReader);
+
+        String str = new String();
+        boolean fLine = true;
+        HashSet<String> t = new HashSet<String>();
+        String listName = "";
+
+        try {
+            while ((str = in.readLine()) != null) {
+                if (fLine && str.length() > 0) {
+                    fLine = false;
+                    listName = str;
+                } else {
+                    t.add(str);
+                }
+            }
+        } catch (IOException ioe) {
+
+        } finally {
+            try {
+                if (in != null) {
+                    in.close();
+                }
+                if (fileReader != null) {
+                    fileReader.close();
+                }
+            } catch (IOException ioe) {
+                ioe.printStackTrace();
+            }
+        }
+
+        if (listName.length() > 0) {
+            HashSet<String> l = m_stopHash.get(listName);
+            if (l != null) {
+                synchronized (l) {
+                    m_stopHash.put(listName, t);
+                }
+            } else {
+                m_stopHash.put(listName, t);
+            }
+        }
+    }
+
+    /**
+     * Is the given word in the stop words list? Uses the defaut "STANDARD"
+     * stoplist
+     * 
+     * @param str
+     *            The word to check
+     * @return is a stop word
+     */
+    public static boolean isStopWord(String str) {
+        boolean retVal = false;
+        if (m_stopHash.containsKey(DEFAULT_STOPLIST))
+            retVal = m_stopHash.get(DEFAULT_STOPLIST).contains(str);
+        return retVal;
+    }
+
+    public static boolean isFirstName(String str) {
+        boolean retVal = false;
+        if (m_stopHash.containsKey("FIRST_NAMES"))
+            retVal = m_stopHash.get("FIRST_NAMES").contains(str.toUpperCase());
+        return retVal;
+    }
+
+    public String getRandomFirstName() {
+        HashSet<String> firstNames = m_stopHash.get("FIRST_NAMES");
+        int indexRand = (int) (Math.random() * new Float(firstNames.size()));
+        Iterator iter = firstNames.iterator();
+        for (int i = 0; i < indexRand; i++) {
+            iter.next();
+        }
+        return ((String) iter.next()).toLowerCase();
+    }
+
+    public static boolean isCommonWord(String str) {
+        if (str == null)
+            return true;
+        String stemmed="";
+		try {
+			stemmed = stemmer.stem(str).toLowerCase();
+		} catch (Exception e) {
+			//stemming exceptions are not informative, jiust ignore wthis word
+			//e.printStackTrace();
+		}
+
+        boolean retVal = false;
+        if (m_stopHash.containsKey("ENG_DICT"))
+            retVal = m_stopHash.get("ENG_DICT").contains(stemmed);
+        return retVal;
+    }
+
+    public boolean isCommonEventWord(String str) {
+        if (str == null)
+            return true;
+        boolean retVal = false;
+
+        try {
+            String stemmed = str.toLowerCase();
+
+            if (m_stopHash.containsKey("fREQUENTEVENTNAMEWORDS"))
+                retVal = m_stopHash.get("fREQUENTEVENTNAMEWORDS").contains(
+                        stemmed);
+        } catch (Exception e) {
+            // TODO Auto-generated catch block
+            e.printStackTrace();
+        }
+        return retVal;
+    }
+
+    /**
+     * Is the given word in the stop words list provided?
+     * 
+     * @param str
+     *            The word to check
+     * @param stop_list
+     *            the name of the stoplist to check against
+     * @return is a stop word
+     */
+    public static boolean isStopWord(String str, String stop_list) {
+        boolean retVal = false;
+        if (m_stopHash.containsKey(stop_list))
+            retVal = m_stopHash.get(stop_list).contains(str);
+        return retVal;
+    }
+
+    public boolean isStopWordAll(String str) {
+        return isStopWord(str);
+    }
+
+    public HashSet<String> getStopListMap(String name) {
+        return m_stopHash.get(name);
+    }
+
+    public static List<List<String>> preFilterCommonEnglishExpressions(
+            List<String> userLikes) {
+        List<List<String>> results = new ArrayList<List<String>>();
+
+        List<String> resultUserLikes = new ArrayList<String>(), potentialCategs = new ArrayList<String>();
+        if (userLikes.size() < 6) {// too short, do not filter
+            results.add(userLikes);
+            results.add(potentialCategs);
+            return results;
+
+        }
+
+        for (String like : userLikes) {
+            like = like.toLowerCase();
+            if (!StringUtils.isAlphanumeric(like.replace(" ", ""))) {
+                logger.info("removed isAlphanumeric " + like);
+                continue;
+            }
+
+            if (StringUtils.isNumeric(like)) {
+                logger.info("removed isNumericSpace " + like);
+                continue;
+            }
+
+            if (like.length() < 4) {
+                logger.info("removed too short likes " + like);
+                continue;
+            }
+            boolean existFirstName = false, allWordsCommonEnglish = true, bStop = false;
+            String[] comps = like.split(" ");
+            StringBuffer buf = new StringBuffer();
+            for (String word : comps) {
+                boolean isCommon = isCommonWord(word);
+                boolean isName = isFirstName(word);
+                if (!isCommon)
+                    allWordsCommonEnglish = false;
+                if (isName)
+                    existFirstName = true;
+                if (isStopWord(word) || word.length() < 3)
+                    bStop = true;
+                else
+                    buf.append(word + " ");
+            } // / does not have to include stop word
+            if (!existFirstName && allWordsCommonEnglish && comps.length < 3) {
+                logger.info("moved to category:  NoFirstName+AllCommonEng+ShorterThan3 "
+                        + like);
+
+                continue;
+            }
+            if (!existFirstName && allWordsCommonEnglish && comps.length == 1) {
+                logger.info("moved to category: NoFirstName+AllCommonEng+Short1word "
+                        + like);
+                potentialCategs.add(like);
+                continue;
+            }
+
+            if (existFirstName && comps.length == 1) {
+                logger.info("removed : only first name, no last name " + like);
+
+                continue;
+            }
+
+            resultUserLikes.add(buf.toString().trim());
+
+        }
+
+        resultUserLikes = new ArrayList<String>(new HashSet<String>(
+                resultUserLikes));
+        if (resultUserLikes.size() > 1) {
+            results.add(resultUserLikes);
+            results.add(potentialCategs);
+            return results;
+        }
+
+        else {// do not do reduction
+            results.add(userLikes);
+            results.add(potentialCategs);
+            return results;
+        }
+    }
+
+    public static boolean isAcceptableIndividualLikes(String like) {
+        StopList finder = StopList.getInstance();
+        like = like.toLowerCase();
+        if (!StringUtils.isAlphanumeric(like.replace(" ", ""))) {
+            logger.info("removed isAlphanumeric " + like);
+            return false;
+        }
+
+        if (StringUtils.isNumeric(like)) {
+            logger.info("removed isNumericSpace " + like);
+            return false;
+        }
+
+        if (like.length() < 4) {
+            logger.info("removed too short likes " + like);
+            return false;
+        }
+        boolean existFirstName = false, allWordsCommonEnglish = true, bStop = false;
+        String[] comps = like.split(" ");
+        StringBuffer buf = new StringBuffer();
+        for (String word : comps) {
+            boolean isCommon = finder.isCommonWord(word);
+            boolean isName = finder.isFirstName(word);
+            if (!isCommon)
+                allWordsCommonEnglish = false;
+            if (isName)
+                existFirstName = true;
+            if (finder.isStopWord(word) || word.length() < 3)
+                bStop = true;
+            else
+                buf.append(word + " ");
+        } // / does not have to include stop word
+        if (!existFirstName && allWordsCommonEnglish && comps.length < 3) {
+            logger.info("  NoFirstName+AllCommonEng+ShorterThan3 " + like);
+
+            return false;
+        }
+        if (!existFirstName && allWordsCommonEnglish && comps.length == 1) {
+            logger.info(" NoFirstName+AllCommonEng+Short1word " + like);
+
+            return false;
+        }
+
+        if (existFirstName && comps.length == 1) {
+            logger.info("removed : only first name, no last name " + like);
+
+            return false;
+        }
+
+        return true;
+    }
+
+    @SuppressWarnings("all")
+    public static void main(String[] args) {
+
+        StopList list = StopList
+                .getInstance("/Users/borisgalitsky/Documents/workspace/opennlp-similarity/src/test/resources/");
+        Boolean b = list.isCommonWord("demonstration");
+
+        String fname = list.getRandomFirstName();
+
+        b = list.isCommonEventWord("tour");
+        b = list.isCommonEventWord("dance");
+        b = list.isCommonEventWord("salsa");
+        b = list.isCommonEventWord("center");
+        b = list.isCommonEventWord("family");
+
+      
+
+        b = isAcceptableIndividualLikes("forest glen");
+        b = isAcceptableIndividualLikes("drive");
+        b = isAcceptableIndividualLikes("house");
+        b = isAcceptableIndividualLikes("Timothy Kloug");
+        b = isAcceptableIndividualLikes("Mamma Mia");
+
+    }
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/TopicAsOpinionMinerRunner.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/TopicAsOpinionMinerRunner.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/TopicAsOpinionMinerRunner.java
new file mode 100644
index 0000000..f4d56aa
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/TopicAsOpinionMinerRunner.java
@@ -0,0 +1,117 @@
+package opennlp.tools.parse_thicket.opinion_processor;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+
+import opennlp.tools.jsmlearning.ProfileReaderWriter;
+import opennlp.tools.parse_thicket.ParseTreeNode;
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+
+public class TopicAsOpinionMinerRunner {
+	private List<File> queue;
+	private final static String reviewSource = "/Users/bgalitsky/Documents/solr/example/exampledocs/publication_page0.json";
+	NamedEntityExtractor neExtractor = new NamedEntityExtractor();
+	Set<String> allPhrases = new HashSet<String>();
+	
+	public void processJSONfileWithReviews(){
+		List<String[]> report = new ArrayList<String[]>();
+		report.add(new String[] { "text", "phrases of potential interest list" , });
+
+		
+		String content=null;
+		try {
+			content = FileUtils.readFileToString(new File(reviewSource));
+		} catch (IOException e) {
+			e.printStackTrace();
+		}
+		String[] texts = StringUtils.substringsBetween(content, "summary\":\"", "\"");
+		for(String text: texts){
+			report.clear();
+			EntityExtractionResult result = neExtractor.extractEntities(text);
+			//report.add(new String[]{text});
+			allPhrases.addAll(result.extractedNERWords);
+			allPhrases = new HashSet<String>(allPhrases);
+			for(String p: allPhrases){
+				report.add(new String[]{p});
+			}
+			/*
+			String[] phrases = (String[])result.extractedNERWords.toArray(new String[0]);
+			if (phrases!=null && phrases.length>0)
+				report.add(phrases);
+			*/
+			/*report.add((String[])result.extractedSentimentPhrases.toArray(new String[0]));
+			List<String> stringPhrases = new ArrayList<String>(),
+					nodePhrases = new ArrayList<String>();
+			for(List<ParseTreeNode> chList: result.extractedSentimentPhrases){
+				String buf = "", nodeBuf="";
+				for(ParseTreeNode ch: chList){
+					buf+=ch.getWord()+ " ";
+					nodeBuf+=ch.toString()+ " ";
+				}
+				stringPhrases.add(buf.trim());
+				nodePhrases.add(nodeBuf.trim());
+			}
+			report.add((String[])stringPhrases.toArray(new String[0]));
+			report.add((String[])nodePhrases.toArray(new String[0]));
+			*/
+			
+			ProfileReaderWriter.writeReport(report, "phrasesExtracted3.csv");
+		}
+	}
+
+	private void addFiles(File file) {
+
+		if (!file.exists()) {
+			System.out.println(file + " does not exist.");
+
+			if (file.isDirectory()) {
+				for (File f : file.listFiles()) {
+					if (f.getName().startsWith("."))
+						continue;
+					addFiles(f);
+					System.out.println(f.getName());
+				}
+			} else {
+				queue.add(file);
+
+			}
+		}
+	}
+	
+	public static void main(String[] args){
+		TopicAsOpinionMinerRunner runner = new TopicAsOpinionMinerRunner();
+		runner.processJSONfileWithReviews();
+
+	}
+}
+
+/*
+	public void processDirectory(String path){
+		List<String[]> report = new ArrayList<String[]>();
+		report.add(new String[] { "filename", "named entity list", "phrases of potential interest list" });
+
+		List<String> allNamedEntities = new ArrayList<String>();
+
+		addFiles(new File(path));
+		for(File f: queue){
+			List<String> entities = (List<String>) extractEntities(f.getAbsolutePath()).getFirst();
+			List<String> opinions = (List<String>) extractEntities(f.getAbsolutePath()).getSecond();
+			report.add(new String[]{ f.getName(), entities.toString(),  opinions.toString()});	
+			ProfileReaderWriter.writeReport(report, "nameEntitiesExtracted.csv");
+
+			allNamedEntities.addAll(entities);
+
+			allNamedEntities = new ArrayList<String>(new HashSet<String> (allNamedEntities ));
+
+
+		}
+		ProfileReaderWriter.writeReport(report, "nameEntitiesTopicsOfInterestExtracted.csv");
+	} 
+} */

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/TopicPhraseExtractor.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/TopicPhraseExtractor.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/TopicPhraseExtractor.java
new file mode 100644
index 0000000..a704f22
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/TopicPhraseExtractor.java
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.parse_thicket.opinion_processor;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import opennlp.tools.apps.relevanceVocabs.SentimentVocab;
+import opennlp.tools.parse_thicket.ParseThicket;
+import opennlp.tools.parse_thicket.ParseTreeNode;
+import opennlp.tools.parse_thicket.matching.Matcher;
+
+public class TopicPhraseExtractor {
+	Matcher matcher = new Matcher();
+
+	// sentiment vocabulary for phrase under the focus of sentiment
+	SentimentVocab sVocab = SentimentVocab.getInstance();
+	//This is used to create an XML with phrases. The same class for acro  & phrases
+
+	public EntityExtractionResult extractEntities(String para){
+		EntityExtractionResult result = new EntityExtractionResult();
+		List<String> extractedNerPhrasesStr = new ArrayList<String>(), 
+				extractedNerExactStr = new ArrayList<String>(),
+				extractedSentimentPhrasesStr = 
+				new ArrayList<String>(), extractedNONSentimentPhrasesStr = 
+				new ArrayList<String>(), extractedNerPhraseTags = new ArrayList<String>();
+		// no need to change to extract more/less phrases
+		ParseThicket pt = matcher.buildParseThicketFromTextWithRST(para);
+
+		List<List<ParseTreeNode>> extractedSentimentPhrases = new ArrayList<List<ParseTreeNode>>(), 
+				extractedNONSentimentPhrases = new ArrayList<List<ParseTreeNode>>(),
+				extractedNerPhrases = new ArrayList<List<ParseTreeNode>>(),
+						extractedNerExactPhrases= new ArrayList<List<ParseTreeNode>>();
+		//TODO document examples / cases for each rule
+		// now we extract phrases
+		List<List<ParseTreeNode>> phrases = pt.getPhrases();
+		List<Float> sentimentProfile = pt.getSentimentProfile();
+		for(List<ParseTreeNode> phrase: phrases){
+
+			// find a noun phrase under sentiment
+			boolean bAccept = true, bNER = false;
+
+			String phraseStr = asString(phrase);
+
+
+			if (!phrase.get(0).getPhraseType().equals("NP") && !phrase.get(0).getPhraseType().equals("VP") )	
+				bAccept = false;
+
+			boolean bSentiment = false;
+			for(ParseTreeNode word: phrase){
+				if (sVocab.isSentimentWord(word.getWord())){
+					bSentiment=true;
+					break;
+				}
+			}
+
+			String nerTagConfirmed = null;
+			for(ParseTreeNode word: phrase){
+				// no Named Entity
+				String nerTag = isNERforPhraseExtraction(word);
+				if (nerTag!=null){
+					bNER = true;
+					nerTagConfirmed = nerTag;
+				}
+
+				// no numbers nor prepositions
+				if (word.getPos().startsWith("CD") || word.getPos().indexOf("PRP")>-1 )
+					bAccept = false;
+			}
+			if (!bAccept)
+				continue;
+			// was 7 -> 2
+			if (phrase.size()>7 || phrase.size()<2)
+				bAccept = false;
+
+			if (phrase.get(0).getPos().equals("DT") && phrase.size()<3)
+				bAccept = false;
+			if (!bAccept)
+				continue;
+
+			String cleanedPhraseStr = cleanPhraseString(phraseStr);
+			if (cleanedPhraseStr==null)
+				bAccept = false;
+
+			if (bAccept){
+				if (bNER){
+					extractedNerPhrases.add(phrase);
+					extractedNerPhrasesStr.add(phraseStr);
+					extractedNerPhraseTags.add(nerTagConfirmed );
+					// forming exact NER
+					List<ParseTreeNode> phraseNER_exact = new ArrayList<ParseTreeNode>();
+					String nerExactStr = "";
+					for(ParseTreeNode word: phrase){
+						String ner = isNERforPhraseExtraction(word);
+						if (ner!=null && ner.equals(nerTagConfirmed)){
+							phraseNER_exact.add(word);
+							nerExactStr+=" "+word.getWord();
+						}
+					}
+					nerExactStr.trim();
+					extractedNerExactPhrases.add(phraseNER_exact);
+					extractedNerExactStr.add(nerExactStr);
+				}
+				else if (bSentiment) {
+					extractedSentimentPhrasesStr.add(cleanedPhraseStr);					
+					extractedSentimentPhrases.add(phrase);
+				} else {
+					extractedNONSentimentPhrasesStr.add(cleanedPhraseStr);					
+					extractedNONSentimentPhrases.add(phrase);
+				}
+			}
+		} 
+
+		result.setExtractedSentimentPhrases(extractedSentimentPhrases);
+		result.setExtractedSentimentPhrasesStr(extractedSentimentPhrasesStr);
+
+		result.setExtractedNONSentimentPhrases(extractedNONSentimentPhrases);
+		result.setExtractedNONSentimentPhrasesStr(extractedNONSentimentPhrasesStr);
+		
+		result.setExtractedNerPhrases(extractedNerPhrases);
+		result.setExtractedNerPhrasesStr(extractedNerPhrasesStr);
+		result.setExtractedNerPhraseTags(extractedNerPhraseTags);
+		
+		result.setExtractedNerExactPhrases(extractedNerExactPhrases);
+		result.setExtractedNerExactStr(extractedNerExactStr);
+
+		result.setSentimentProfile(sentimentProfile );
+
+		return result;
+	}
+
+
+
+
+
+
+	private String cleanPhraseString(String phraseStr) {
+		String p = phraseStr.toLowerCase();
+
+		if (p.startsWith("*") || p.startsWith("&") || p.startsWith("$"))
+			return null;
+
+		if (p.startsWith("this ") || p.startsWith("other "))
+			return null;
+
+		if (p.startsWith("a "))
+			p = p.substring(2, p.length());
+		if (p.startsWith("the "))
+			p = p.substring(4, p.length());
+		if (p.startsWith(", "))
+			p = p.substring(2, p.length());
+
+		return p;
+	}
+
+	private String asString(List<ParseTreeNode> phrase) {
+		String buf = "";
+		for(ParseTreeNode p: phrase)
+			buf+=p.getWord()+" ";
+		return buf.trim();
+	}
+
+	private String isNERforPhraseExtraction(ParseTreeNode word){
+		if (word.getNe() == null)
+			return null;
+		
+
+		if (!(word.getPos().startsWith("NN") || word.getPos().startsWith("PR") || word.getPos().startsWith("IN")|| 
+				word.getPos().startsWith("JJ") || word.getPos().startsWith("DT")))
+			return null;
+				
+
+		if (word.getNe().equals("ORGANIZATION"))
+				return "ORGANIZATION";
+		if(word.getNe().equals("LOCATION"))
+			return "LOCATION";
+					
+		if(word.getNe().equals("PERSON") ) 
+			return "PERSON";
+		
+		if(word.getNe().equals("MONEY") ) 
+			return "MONEY";
+		if(word.getNe().equals("DATE") ) 
+			return "DATE";
+		if(word.getNe().equals("TIME") ) 
+			return "TIME";
+
+		return null;
+
+	}
+}
+
+/*
+ * Na�ve  sentiment prediction systems work just by looking at words in isolation, giving positive points for positive words and negative points for negative words and then summing up these points. That way, the order of words is ignored and important information is lost. The deep learning model of (Socher et al 2013) builds a representation of whole sentences based on the sentence structure. It computes the sentiment based on how words compose the meaning of longer phrases. However, in most applications just taking individual sentences into account do not give accurate results and rhetoric information needs to be taken into account to determine the overall sentiment of a paragraph and then back to the individual sentence level.
+ */
+

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/TwitterEngineRunner.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/TwitterEngineRunner.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/TwitterEngineRunner.java
new file mode 100644
index 0000000..6de3180
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/TwitterEngineRunner.java
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.parse_thicket.opinion_processor;
+
+import java.io.File;
+import java.io.IOException;
+import java.lang.reflect.Array;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+
+import au.com.bytecode.opencsv.CSVWriter;
+import opennlp.tools.jsmlearning.ProfileReaderWriter;
+import opennlp.tools.parse_thicket.ParseTreeNode;
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+
+public class TwitterEngineRunner {
+	private List<File> queue;
+	private final static String twSource = "/Users/bgalitsky/Documents/workspace/TwitterMiner/data/TwitterArtistsDynamicsTot12_07.csv";
+	TwitterFilter neExtractor = new TwitterFilter();
+	private static int iWind = 80;
+
+	public void processTweetFile(int nRun){
+		List<String[]> report = new ArrayList<String[]>(), ful_less =  new ArrayList<String[]>();
+		List<String> meaningLESS = new ArrayList<String>(), meaningFUL = new ArrayList<String>();
+		report.add(new String[] { "text", "phrases of potential interest list" , });
+
+		List<String[]> texts = ProfileReaderWriter.readProfiles(twSource);
+		int offset = iWind*nRun;
+		
+		//for(int i=offset; i< offset+iWind; i++){
+			
+		//	String[] text = texts.get(i);
+		for(String[] text: texts){
+			List<String> textDeduped = new ArrayList<String>(new HashSet<String>(Arrays.asList(text)));
+			EntityExtractionResult result = null;
+			if (text==null || text.length<4)
+				continue;
+
+			for(int nInLine=3; nInLine<textDeduped.size(); nInLine++){
+				if (textDeduped.get(nInLine).length()>180)
+					continue;
+				
+				String cleanedTweet = textDeduped.get(nInLine).replace("/\\bs\\@+/ig","");
+				try {
+					result = neExtractor.extractEntities(cleanedTweet);
+				} catch (Exception e) {
+					e.printStackTrace();
+					continue;
+				}
+				report.add(new String[]{text[0],text[nInLine]});
+				report.add((String[])result.extractedNERWords.toArray(new String[0]));
+				//report.add((String[])result.extractedSentimentPhrases.toArray(new String[0]));
+				List<String> stringPhrases = new ArrayList<String>(),
+						nodePhrases = new ArrayList<String>();
+				Boolean bMeaningf = false;
+
+				//stringPhrases.add(""); nodePhrases.add(""); // to make report more readable
+				for(List<ParseTreeNode> chList: result.extractedSentimentPhrases){
+					String buf = "", nodeBuf="";
+					for(ParseTreeNode ch: chList){
+						buf+=ch.getWord()+ " ";
+						nodeBuf+=ch.toString()+ " ";
+					}
+					stringPhrases.add(buf.trim());
+					nodePhrases.add(nodeBuf.trim());
+				}
+				// selecting MEANINGFULL
+				if (nodePhrases.size()>1){
+					if ((nodePhrases.get(0).indexOf(">VP'")>-1 || nodePhrases.get(0).indexOf(">NNP'")>-1) &&
+							(nodePhrases.get(1).indexOf(">VP'")>-1 || nodePhrases.get(1).indexOf(">NNP'")>-1)){
+						bMeaningf = true;
+
+					}
+				}
+
+				report.add((String[])stringPhrases.toArray(new String[0]));
+				report.add((String[])nodePhrases.toArray(new String[0]));
+				if (bMeaningf){
+					report.add(new String[]{"===", "MEANINGFUL tweet"});
+					if (!meaningFUL.contains(cleanedTweet))
+						meaningFUL.add(cleanedTweet);
+				} else {
+					if (!meaningLESS.contains(cleanedTweet))
+						meaningLESS.add(cleanedTweet);
+				}
+
+				int count = 0;
+				ful_less.clear();
+				for(String less: meaningLESS ){
+					String fl = "";
+					if (count<meaningFUL.size())
+						fl = meaningFUL.get(count);
+					ful_less.add(new String[]{less, fl});
+					count++;
+				}
+
+				report.add(new String[]{"-----------------------------------------------------"});
+					ProfileReaderWriter.writeReport(report, "phrasesExtractedFromTweets3_"+nRun+".csv");
+					ProfileReaderWriter.writeReport(ful_less, "ful_lessTweets3_"+nRun+".csv");
+				
+			}
+		}
+	}
+
+
+	public static void main(String[] args){
+		TwitterEngineRunner runner = new TwitterEngineRunner();
+		int nRun = Integer.parseInt(args[0]);
+		runner.processTweetFile(nRun);
+
+	}
+}
+
+/*
+	public void processDirectory(String path){
+		List<String[]> report = new ArrayList<String[]>();
+		report.add(new String[] { "filename", "named entity list", "phrases of potential interest list" });
+
+		List<String> allNamedEntities = new ArrayList<String>();
+
+		addFiles(new File(path));
+		for(File f: queue){
+			List<String> entities = (List<String>) extractEntities(f.getAbsolutePath()).getFirst();
+			List<String> opinions = (List<String>) extractEntities(f.getAbsolutePath()).getSecond();
+			report.add(new String[]{ f.getName(), entities.toString(),  opinions.toString()});	
+			ProfileReaderWriter.writeReport(report, "nameEntitiesExtracted.csv");
+
+			allNamedEntities.addAll(entities);
+
+			allNamedEntities = new ArrayList<String>(new HashSet<String> (allNamedEntities ));
+
+
+		}
+		ProfileReaderWriter.writeReport(report, "nameEntitiesTopicsOfInterestExtracted.csv");
+	} 
+} */
+
+

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/TwitterFilter.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/TwitterFilter.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/TwitterFilter.java
new file mode 100644
index 0000000..0e5053d
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/TwitterFilter.java
@@ -0,0 +1,165 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.parse_thicket.opinion_processor;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+
+import opennlp.tools.apps.relevanceVocabs.SentimentVocab;
+import opennlp.tools.jsmlearning.ProfileReaderWriter;
+import opennlp.tools.parse_thicket.ParseThicket;
+import opennlp.tools.parse_thicket.ParseTreeNode;
+import opennlp.tools.parse_thicket.VerbNetProcessor;
+import opennlp.tools.parse_thicket.kernel_interface.DescriptiveParagraphFromDocExtractor;
+import opennlp.tools.parse_thicket.matching.Matcher;
+import opennlp.tools.parse_thicket.matching.PT2ThicketPhraseBuilder;
+import opennlp.tools.similarity.apps.utils.Pair;
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+import opennlp.tools.textsimilarity.TextProcessor;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang.StringUtils;
+
+public class TwitterFilter {
+	protected static Matcher matcher;
+	private static int PARA_LENGTH_IN_SENTENCES = 5, PARA_LENGTH = 250;
+	protected ArrayList<File> queue = new ArrayList<File>();
+	protected static PT2ThicketPhraseBuilder phraseBuilder;
+	protected static SentimentVocab sVocab = SentimentVocab.getInstance();
+	String resourceDirSentimentList = null;
+	Set<String> sentimentVcb = new HashSet<String> ();
+
+	static {
+		synchronized (TwitterFilter.class) {
+			matcher = new Matcher();
+			phraseBuilder = new PT2ThicketPhraseBuilder();
+		}
+	}
+
+	public TwitterFilter(){
+		try {
+			resourceDirSentimentList = new File( "." ).getCanonicalPath()+"/src/test/resources/opinions/sentiment_listReduced.csv";
+		} catch (IOException e) {
+			e.printStackTrace();
+		}
+		List<String[]> sentimentList=null;
+		sentimentList = ProfileReaderWriter.readProfiles(resourceDirSentimentList);
+		for(String[] line: sentimentList){
+			sentimentVcb.add(line[0]);
+		}
+	}
+
+	private boolean isSentimentWord(String word){
+		if (sentimentVcb.contains(word))
+			return true;
+		else
+			return false;		
+	}
+
+	public EntityExtractionResult extractEntities(String para){
+		List<List<ParseTreeNode>> extractedNERs = new ArrayList<List<ParseTreeNode>>();
+		List<String> extractedNERsWords = new ArrayList<String>();
+		List<List<ParseTreeNode>> extractedSentimentPhrases = 
+				new ArrayList<List<ParseTreeNode>>();
+		EntityExtractionResult result = new EntityExtractionResult();
+
+		ParseThicket pt = null;
+
+		System.out.println("Processing paragraph of length "+para.length() + " | "+ para);
+		pt = matcher.buildParseThicketFromTextWithRST(para);
+		List<List<ParseTreeNode>> nodeList = pt.getSentenceNodes();
+
+
+		for(List<ParseTreeNode> sentence: nodeList){
+			System.out.println("   Processing sentence: "+ sentence);
+			boolean bInsideNER = false; 
+			String currentPhrase = "";
+			List<ParseTreeNode> currentPhraseNode = new ArrayList<ParseTreeNode>(); 
+			for(ParseTreeNode word: sentence){
+				if (isNERforPhraseExtraction(word)){
+					System.out.println("++Found word ="+word + " | NER="+ word.getNe());
+					if (bInsideNER){
+						currentPhrase += " "+word.getWord();
+						currentPhraseNode.add(word);
+					} else {
+						bInsideNER=true;
+						currentPhrase = word.getWord();
+						currentPhraseNode.add(word);
+					}
+				} else {
+					if (bInsideNER){
+						if (currentPhrase.indexOf(' ')>-1) // at least two tokens
+							extractedNERsWords.add(currentPhrase);
+							extractedNERs.add(currentPhraseNode);
+						currentPhrase = "";
+						bInsideNER=false;
+					} else {
+						// do nothing, continue scan
+					}
+				}
+			}
+			if (currentPhrase.length()>1 && currentPhrase.indexOf(' ')>-1){
+				extractedNERs.add(currentPhraseNode);
+				extractedNERsWords.add(currentPhrase);
+			}
+
+			Set<String> foundSentimentWords = new HashSet<String>();
+			// now we extract phrases
+			List<List<ParseTreeNode>> phrases = pt.getPhrases();
+			for(List<ParseTreeNode> phrase: phrases){
+				// find a noun phrase under sentiment
+				try {
+					for(int i = phrase.size()-1; i>-1; i--){
+						ParseTreeNode word = phrase.get(i);
+						if ((isSentimentWord(word.getWord()) ||
+								sVocab.isSentimentWord(word.getWord()) && !foundSentimentWords.contains(word.getWord()) )){
+							foundSentimentWords.add(word.getWord());
+							System.out.println("Found opinionated phrase "+phrase.toString());
+							extractedSentimentPhrases.add(phrase);			
+							break;
+						}
+					}
+				} catch (Exception e) {
+					e.printStackTrace();
+				}
+			}
+
+		} 
+		result.setExtractedNER(extractedNERs);
+		result.setExtractedNERWords(extractedNERsWords);
+		result.setExtractedSentimentPhrases(extractedSentimentPhrases);
+		return result;
+	}
+
+
+
+	private boolean isNERforPhraseExtraction(ParseTreeNode word){
+		if ((word.getNe().equals("ORGANIZATION") ||word.getNe().equals("LOCATION") || word.getNe().equals("PERSON") ) &&
+				(word.getPos().startsWith("NN") || word.getPos().startsWith("PR") || word.getPos().startsWith("IN")|| 
+						word.getPos().startsWith("JJ") || word.getPos().startsWith("DT")  ))
+			return true;
+
+		return false;
+
+	}
+
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/YouTubeMiner.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/YouTubeMiner.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/YouTubeMiner.java
new file mode 100644
index 0000000..a138de6
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/YouTubeMiner.java
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.parse_thicket.opinion_processor;
+
+import java.io.File;
+import java.io.IOException;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang.StringUtils;
+
+import opennlp.tools.similarity.apps.utils.PageFetcher;
+
+public class YouTubeMiner {
+	private PageFetcher fetcher = new PageFetcher();
+	public YouTubeMinerResult getData(String url){
+		YouTubeMinerResult result = new YouTubeMinerResult();
+		String content = fetcher.fetchOrigHTML(url);
+		try {
+			FileUtils.writeStringToFile(new File(url.replace(':', '_').replace('/', '_')), content);
+		} catch (IOException e1) {
+			// TODO Auto-generated catch block
+			e1.printStackTrace();
+		}
+		if (url.indexOf("channel")>-1){
+			try { //subscriber-count" title="30" 
+				String subscribersStr = StringUtils.substringBetween(content,"subscriber-count", "tabindex");
+				String dirtyNumber = StringUtils.substringBetween(subscribersStr, "title=\"", "\"");
+				String cleanNumber = dirtyNumber.replaceAll("[^\\x00-\\x7F]", "");
+				if (cleanNumber!=null){
+					int subscribers = Integer.parseInt(cleanNumber );
+					result.subscribers = subscribers;
+				} else {
+					System.err.println("Not found data for 'subscriber-count', 'tabindex'");
+				}
+			} catch (NumberFormatException e) {
+				// TODO Auto-generated catch block
+				e.printStackTrace();
+			}
+		} else {
+			try {
+
+				String subscribersStr = StringUtils.substringBetween(content,"subscriber-count", "tabindex");
+				String dirtyNumber = StringUtils.substringBetween(subscribersStr, "title=\"", "\"").replace(" ", "");
+				if (dirtyNumber!=null){
+					int subscribers = Integer.parseInt(dirtyNumber );
+					result.subscribers = subscribers;
+				} else {
+					System.err.println("Not found data for 'subscriber-count', 'tabindex'");
+				}
+
+				String viewsStrDirty = StringUtils.substringBetween(content,
+						//"div class=\"watch-view-count\">"," views</div>");
+						//view-count">12 \u043f\u0440\u043e\u0441\u043c\u043e\u0442\u0440\u043e\u0432</div>
+						"view-count","<div>");
+				String viewsStr = StringUtils.substringBetween(viewsStrDirty,">", " ");
+				if (viewsStr!=null){
+					int views = Integer.parseInt(viewsStr );
+					result.views = views;
+				} else {
+					System.err.println("Not found data for 'view-count','<div>'");
+				}
+			} catch (NumberFormatException e) {
+				// TODO Auto-generated catch block
+				e.printStackTrace();
+			}
+		}
+
+		return result;
+	}
+
+
+
+
+	public static void main(String[] args){
+		YouTubeMiner  miner = new YouTubeMiner();
+		System.out.println(miner.getData("https://www.youtube.com/channel/UC-maQbG5eUS5c1wmaTnLwTA"));
+		System.out.println(miner.getData("https://www.youtube.com/watch?v=U6X4VT9dVr8"));
+		System.out.println(miner.getData("https://www.youtube.com/watch?v=kH-AQnta714"));
+		System.out.println(miner.getData("https://www.youtube.com/watch?v=pWb50Kn1ShQ"));
+	}
+}
+
+

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/YouTubeMinerResult.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/YouTubeMinerResult.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/YouTubeMinerResult.java
new file mode 100644
index 0000000..86c8e9d
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/YouTubeMinerResult.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.parse_thicket.opinion_processor;
+
+public class YouTubeMinerResult {
+	public int likes;
+	public int subscribers;
+	public int views;
+	
+	boolean isPromisingYoungIndividual(){
+		if (subscribers>0)
+			if (subscribers>10 && subscribers< 20000)
+				return true;
+		if (views>0)
+			if (views>10 && views< 20000)
+				return true;
+		return false;
+
+	}
+	
+	public String toString(){
+		return "views :"+ views + "| subscribers = "+ subscribers;
+	}
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/pattern_structure/LinguisticPatternStructure.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/pattern_structure/LinguisticPatternStructure.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/pattern_structure/LinguisticPatternStructure.java
new file mode 100755
index 0000000..3c88b41
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/pattern_structure/LinguisticPatternStructure.java
@@ -0,0 +1,192 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.parse_thicket.pattern_structure;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+
+import opennlp.tools.fca.ConceptLattice;
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+
+public class LinguisticPatternStructure extends PhrasePatternStructure {
+
+	public LinguisticPatternStructure(int objectCounts, int attributeCounts) {
+		super(objectCounts, attributeCounts);
+		
+		ConceptLattice cl = null;
+	}
+	
+	public void AddExtentToAncestors(LinkedHashSet<Integer>extent, int curNode) {
+		//
+		if (conceptList.get(curNode).parents.size()>0){
+			for (int parent : conceptList.get(curNode).parents){
+				conceptList.get(parent).addExtents(extent);
+				AddExtentToAncestors(extent, parent);
+			}
+		}	
+	}
+	
+	public int AddIntent(List<List<ParseTreeChunk>> intent, LinkedHashSet<Integer>extent,int generator) {
+		System.out.println("debug");
+		System.out.println("called for " + intent);
+		//printLattice();
+		int generator_tmp = GetMaximalConcept(intent, generator);
+		generator = generator_tmp;
+		if (conceptList.get(generator).intent.equals(intent)) {
+			System.out.println("at generator:" + conceptList.get(generator).intent);
+			System.out.println("to add:" + intent);
+			System.out.println("already generated");
+			AddExtentToAncestors(extent, generator);	
+			return generator;
+		}
+		Set<Integer> generatorParents = conceptList.get(generator).parents;
+		Set<Integer> newParents = new HashSet<Integer>();
+		for (int candidate : generatorParents) {
+			if (!intent.containsAll(conceptList.get(candidate).intent)) {
+				List<List<ParseTreeChunk>> intersection = md
+				.matchTwoSentencesGroupedChunksDeterministic(intent, conceptList.get(candidate).intent);
+				LinkedHashSet<Integer> new_extent = new LinkedHashSet<Integer>();
+				new_extent.addAll(conceptList.get(candidate).extent);
+				new_extent.addAll(extent);
+				if (intent.size()!=intersection.size()){
+					System.out.println("recursive call (inclusion)");
+					System.out.println(intent + "----" + intersection);
+					candidate = AddIntent(intersection,new_extent, candidate);
+				}
+			}
+			
+			boolean addParents = true;
+			System.out.println("now iterating over parents");
+			Iterator<Integer> iterator = newParents.iterator();
+			while (iterator.hasNext()) {
+				Integer parent = iterator.next();
+				if (conceptList.get(parent).intent.containsAll(conceptList.get(candidate).intent)) {
+					addParents = false;
+					break;
+				}
+				else {
+					if (conceptList.get(candidate).intent.containsAll(conceptList.get(parent).intent)) {
+						iterator.remove();
+					}
+				}
+			}
+			if (addParents) {
+				newParents.add(candidate);
+			}
+		}
+		System.out.println("size of lattice: " + conceptList.size());
+		PhraseConcept newConcept = new PhraseConcept();
+		newConcept.setIntent(intent);
+
+		LinkedHashSet<Integer> new_extent = new LinkedHashSet<Integer>();
+		new_extent.addAll(conceptList.get(generator).extent);
+		new_extent.addAll(extent);
+		newConcept.addExtents(new_extent);
+		
+		newConcept.setPosition(conceptList.size());
+		conceptList.add(newConcept);
+		conceptList.get(generator).parents.add(newConcept.position);
+		conceptList.get(newConcept.position).childs.add(generator);
+		for (int newParent: newParents) {
+			if (conceptList.get(generator).parents.contains(newParent)) {
+				conceptList.get(generator).parents.remove(newParent);
+				conceptList.get(newParent).childs.remove(generator);
+			}
+			conceptList.get(newConcept.position).parents.add(newParent);
+			conceptList.get(newParent).addExtents(new_extent);
+			AddExtentToAncestors(new_extent, newParent);
+			conceptList.get(newParent).childs.add(newConcept.position);
+		}
+		return newConcept.position;
+	}
+	
+	public void printLatticeExtended() {
+		for (int i = 0; i < conceptList.size(); ++i) {
+			printConceptByPositionExtended(i);
+		}
+	}
+	
+	public void printConceptByPositionExtended(int index) {
+		System.out.println("Concept at position " + index);
+		conceptList.get(index).printConceptExtended();
+	}
+	
+	
+	public int [][] toContext(int extentCardinality){
+		
+		int newAttrCount = conceptList.size();
+		ArrayList<PhraseConcept> cList = new ArrayList<PhraseConcept>();
+		cList.addAll(conceptList);	
+		boolean run = true;
+		int k=0;
+		while (run && k<conceptList.size()){
+			if (conceptList.get(k).intent.size() == attributeCount){
+				if (conceptList.get(k).extent.size() == 0)
+					for (Integer i:conceptList.get(k).parents)
+						cList.remove(i);
+				cList.remove(k);
+				run=false;
+			}
+			else
+				k+=1;	
+		}
+		
+		run = true;
+		k=0;
+		while (run && k<=newAttrCount){
+			if (cList.get(k).extent.size()==0)
+				k++;
+				run = false;
+		}
+		newAttrCount = cList.size();
+		Set<Integer> nodeExtend;
+		int [][] binaryContext = new int[extentCardinality][newAttrCount];
+		for (int j = 0; j<newAttrCount; j++){
+			nodeExtend = cList.get(j).extent;
+			for (Integer i: nodeExtend){
+				binaryContext[i][j]=1;
+			}
+		}
+		return binaryContext;
+	}
+	
+	
+	
+	public void logStability(){
+		int min_delta = -1, delta = -1;
+		float sum = 0;
+		for (int i = 0; i < conceptList.size(); ++i) {
+			min_delta = Integer.MAX_VALUE;
+			sum = 0;
+			PhraseConcept pc = conceptList.get(i);
+			Set<Integer> childs = pc.childs;
+			for (Integer j: childs) {
+				delta = pc.extent.size() - conceptList.get(j).extent.size();
+				if (delta<min_delta)
+					min_delta = delta;
+				sum += Math.pow(2, -delta);
+			}
+			pc.intLogStabilityBottom=-(Math.log(sum)/Math.log(2.0));
+			pc.intLogStabilityUp = min_delta;
+		}
+	}
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/pattern_structure/PatternStructureWriter.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/pattern_structure/PatternStructureWriter.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/pattern_structure/PatternStructureWriter.java
new file mode 100755
index 0000000..c3f5688
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/pattern_structure/PatternStructureWriter.java
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.parse_thicket.pattern_structure;
+
+import java.io.BufferedWriter;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.Writer;
+
+import opennlp.tools.fca.ConceptLattice;
+import opennlp.tools.fca.FormalConcept;
+
+public class PatternStructureWriter {
+	
+	public void WriteStatsToTxt(String filename, PhrasePatternStructure ps){
+			
+		String formatStr = "[%5.2f; %5.2f]  %s   %s%n";
+		Writer writer = null;
+
+		try {
+		    writer = new BufferedWriter(new OutputStreamWriter(
+		          new FileOutputStream(filename), "utf-8"));
+		    writer.write("PatternStructure size: " + ps.conceptList.size()+ " with " + ps.objectCount + "objects\n");
+		    
+		    for (PhraseConcept c : ps.conceptList){
+		    	writer.write(String.format(formatStr,c.intLogStabilityBottom, c.intLogStabilityUp, c.extent, c.intent));
+		    }
+		    writer.close();
+		    
+		} catch (IOException ex) {
+			System.err.println(ex.getMessage());
+		} finally {
+		   try {writer.close();} catch (Exception ex) {}
+		}
+	}
+	
+
+		public static void main(String[] args) {
+			
+		}
+}
+			
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/request_response_recognizer/TreeKernelBasedRecognizerOfRequest_Response.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/request_response_recognizer/TreeKernelBasedRecognizerOfRequest_Response.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/request_response_recognizer/TreeKernelBasedRecognizerOfRequest_Response.java
new file mode 100644
index 0000000..e33e089
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/request_response_recognizer/TreeKernelBasedRecognizerOfRequest_Response.java
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.parse_thicket.request_response_recognizer;
+
+
+import java.util.ArrayList;
+import java.util.List;
+
+import opennlp.tools.parse_thicket.ParseThicket;
+import opennlp.tools.parse_thicket.VerbNetProcessor;
+import opennlp.tools.parse_thicket.external_rst.MatcherExternalRST;
+import opennlp.tools.parse_thicket.external_rst.ParseThicketWithDiscourseTree;
+import opennlp.tools.parse_thicket.kernel_interface.TreeKernelBasedClassifierMultiplePara;
+
+/*
+ * This class performs TK learning based on parse thicket which includes RST relations only 
+ * based on Surdeanu at al RST parser. It does sentence parsing and NLP pipeline of 
+ * Surdeanu's wrapper of Stanford NLP
+ */
+public class TreeKernelBasedRecognizerOfRequest_Response extends TreeKernelBasedClassifierMultiplePara{
+
+	private MatcherExternalRST matcherRST = new MatcherExternalRST();
+
+	protected List<String> formTreeKernelStructuresMultiplePara(List<String> texts, String flag) {
+		//TODO
+		this.setShortRun();	
+		List<String> extendedTreesDumpTotal = new ArrayList<String>();
+		try {
+
+			for(String text: texts){
+				// get the parses from original documents, and form the training dataset
+				try {
+					System.out.print("About to build pt with external rst from "+text + "\n...");
+					ParseThicket pt = matcherRST.buildParseThicketFromTextWithRST(text);
+					if (pt == null)
+						continue;
+					System.out.print("About to build extended forest with external rst...");
+					List<String> extendedTreesDump =  // use direct option (true
+							buildReptresentationForDiscourseTreeAndExtensions((ParseThicketWithDiscourseTree)pt, true);
+					for(String line: extendedTreesDump)
+						extendedTreesDumpTotal.add(flag + " |BT| "+line + " |ET| ");
+					System.out.println("DONE");
+				} catch (Exception e) {
+					e.printStackTrace();
+				}
+			}
+		} catch (Exception e) {
+			e.printStackTrace();
+		}
+		return extendedTreesDumpTotal;
+	}
+
+	private List<String> buildReptresentationForDiscourseTreeAndExtensions(ParseThicketWithDiscourseTree pt, boolean bDirectDT){
+		List<String> extendedTreesDump = new ArrayList<String>();
+		if (!bDirectDT)
+			// option 1: use RST relation for extended trees 
+			extendedTreesDump = treeExtender.buildForestForRSTArcs(pt);
+		else {
+			// option 2: use DT directly
+			extendedTreesDump.add(pt.getDtDump());
+		    extendedTreesDump.add(pt.getDtDumpWithPOS());
+		    extendedTreesDump.add(pt.getDtDumpWithEmbeddedTrees());
+		    extendedTreesDump.add(pt.getDtDumpWithVerbNet());
+		}		
+		return extendedTreesDump;
+	}
+	
+	public static void main(String[] args){
+		VerbNetProcessor p = VerbNetProcessor.
+				getInstance("/Users/bgalitsky/Documents/relevance-based-on-parse-trees/src/test/resources"); 
+
+		TreeKernelBasedRecognizerOfRequest_Response proc = new TreeKernelBasedRecognizerOfRequest_Response();
+		proc.setKernelPath("/Users/bgalitsky/Documents/relevance-based-on-parse-trees/src/test/resources/tree_kernel/");
+		proc.trainClassifier(
+				YahooAnswersTrainingSetCreator.origFilesDir,
+				YahooAnswersTrainingSetCreator.origFilesDir.replace("/text", "/neg_text")
+				);
+	}
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/request_response_recognizer/YahooAnswersTrainingSetCreator.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/request_response_recognizer/YahooAnswersTrainingSetCreator.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/request_response_recognizer/YahooAnswersTrainingSetCreator.java
new file mode 100644
index 0000000..c060c95
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/request_response_recognizer/YahooAnswersTrainingSetCreator.java
@@ -0,0 +1,118 @@
+package opennlp.tools.parse_thicket.request_response_recognizer;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import opennlp.tools.similarity.apps.BingQueryRunner;
+
+import org.apache.commons.io.FileUtils;
+
+public class YahooAnswersTrainingSetCreator {
+	protected List<File> queuePos = new ArrayList<File>(), queueNeg = new ArrayList<File>();
+	public static String origFilesDir = "/Users/bgalitsky/Downloads/NewCategoryIdentification/text";
+	//private BingQueryRunner searcher = new BingQueryRunner();
+	protected void addFilesPos(File file) {
+
+		if (!file.exists()) {
+			System.out.println(file + " does not exist.");
+		}
+		if (file.isDirectory()) {
+			for (File f : file.listFiles()) {
+				addFilesPos(f);
+				System.out.println(f.getName());
+			}
+		} else {
+			queuePos.add(file);
+		}
+	}
+	
+	protected void addFilesNeg(File file) {
+
+		if (!file.exists()) {
+			System.out.println(file + " does not exist.");
+		}
+		if (file.isDirectory()) {
+			for (File f : file.listFiles()) {
+				addFilesNeg(f);
+				System.out.println(f.getName());
+			}
+		} else {
+			queueNeg.add(file);
+		}
+	}
+	
+	public void formNegTrainingSet(String posPath , String negPath){
+		 if (!new File(negPath).exists())
+			 new File(negPath).mkdir();
+		 
+		addFilesPos(new File(posPath));
+		for(int i=0; i< queuePos.size()-1; i+=2){ //take two files at a time
+			File f1 = queuePos.get(i), f2 = queuePos.get(i+1);
+			String content1 = null, content2 = null;
+            try {
+	            content1 = FileUtils.readFileToString(f1);
+	            content2 = FileUtils.readFileToString(f2);
+            } catch (IOException e) {
+	            e.printStackTrace();
+            }
+			String[] portions1 = content1.split("\n\n");
+			String[] portions2 = content2.split("\n\n");
+
+			portions1 = splitIntoRR(portions1, content1);
+			portions2 = splitIntoRR(portions2, content2);
+			if (portions1==null || portions2==null)
+				continue;
+			// do cross-breeding
+			try {
+	            FileUtils.writeStringToFile(new File(negPath+"/" + f1.getName()+".txt"),
+	            		portions1[0] + "\n\n" + portions2[1] );
+	            FileUtils.writeStringToFile(new File(negPath+"/" + f2.getName()+".txt"),
+	            		portions2[0] + "\n\n" + portions1[1] );
+            } catch (IOException e) {
+	            e.printStackTrace();
+            }
+		}
+		
+		
+	}
+	private String[] splitIntoRR(String[] portions, String content) {
+		if (portions.length<2 ){
+			portions = content.replace("?","#_#").split("#_#");
+		}
+		if (portions.length<2 ){
+			portions = content.split("\n");
+		}
+		if (portions.length<2)
+			return null;
+		if (portions.length>2){
+			String q= "", a = "";
+			boolean bQ = true;
+			for(int p=0; p<portions.length; p++){
+				if ( bQ )
+					q+=portions[p]+" \n";
+				else
+					a +=portions[p]+" \n";
+				
+				if (portions[p].endsWith("?")){
+					bQ=false;
+				}
+
+			}
+			if (!bQ) {
+				portions = new String[2];
+				portions[0] = q;
+				portions[1] = a;
+			} else
+				return null;
+		}
+		
+		return portions;
+    }
+	
+	public static void main(String[] args){
+		String dir = YahooAnswersTrainingSetCreator.origFilesDir;
+		new YahooAnswersTrainingSetCreator().formNegTrainingSet(dir, dir.replace("/text", "/neg_text"));
+	}
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/BingRelatedSpellingQueryRunner.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/BingRelatedSpellingQueryRunner.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/BingRelatedSpellingQueryRunner.java
new file mode 100644
index 0000000..025403c
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/BingRelatedSpellingQueryRunner.java
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.similarity.apps;
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import java.util.ArrayList;
+import java.util.List;
+
+import net.billylieurance.azuresearch.AzureSearchRelatedSearchQuery;
+import net.billylieurance.azuresearch.AzureSearchRelatedSearchResult;
+import net.billylieurance.azuresearch.AzureSearchResultSet;
+import net.billylieurance.azuresearch.AzureSearchSpellingSuggestionQuery;
+import net.billylieurance.azuresearch.AzureSearchSpellingSuggestionResult;
+import net.billylieurance.azuresearch.AzureSearchWebQuery;
+import net.billylieurance.azuresearch.AzureSearchWebResult;
+
+public class BingRelatedSpellingQueryRunner extends BingQueryRunner{
+	private AzureSearchRelatedSearchQuery aq = new AzureSearchRelatedSearchQuery ();
+	private AzureSearchSpellingSuggestionQuery  ssq = new AzureSearchSpellingSuggestionQuery ();
+	
+	
+	public List<HitBase> runSearch(String query, int nRes) {
+		aq.setAppid(BING_KEY);
+		aq.setQuery(query);		
+		aq.setPerPage(nRes);
+		aq.doQuery();
+		
+		List<HitBase> results = new ArrayList<HitBase> ();
+		AzureSearchResultSet<AzureSearchRelatedSearchResult> ars = aq.getQueryResult();
+		
+		for (AzureSearchRelatedSearchResult anr : ars){
+		    HitBase h = new HitBase();
+		    h.setTitle(anr.getTitle());
+		    h.setUrl(anr.getBingUrl());
+		    results.add(h);
+		}
+		return results;
+	}
+	
+	public List<HitBase> runSSSearch(String query, int nRes) {
+		ssq.setAppid(BING_KEY);
+		ssq.setQuery(query);		
+		ssq.setPerPage(nRes);
+		ssq.doQuery();
+		
+		List<HitBase> results = new ArrayList<HitBase> ();
+		AzureSearchResultSet<AzureSearchSpellingSuggestionResult> ars = ssq.getQueryResult();
+		
+		for ( AzureSearchSpellingSuggestionResult anr : ars){
+		    HitBase h = new HitBase();
+		    h.setTitle(anr.getTitle());
+		    h.setAbstractText(anr.getValue());
+		   
+		    results.add(h);
+		}
+		return results;
+	}
+	
+	public static void main(String[] args) {
+		BingRelatedSpellingQueryRunner self = new BingRelatedSpellingQueryRunner();
+	    try {
+	    	self.setLang("es-MX");
+	    	self.setKey(
+	    			"e8ADxIjn9YyHx36EihdjH/tMqJJItUrrbPTUpKahiU0=");
+	      List<HitBase> resp = self
+	          .runSearch("clear Sess", 10);
+	      System.out.print(resp.get(0));
+	      
+	      resp = self
+		          .runSSSearch("clear Sess", 10);
+		      System.out.print(resp.get(0));
+	    } catch (Exception e) {
+	      // TODO Auto-generated catch block
+	      e.printStackTrace();
+	    }
+	}
+}


[30/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/70FictWildeO_Dorian_18_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/70FictWildeO_Dorian_18_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/70FictWildeO_Dorian_18_EN.txt.txt
new file mode 100644
index 0000000..6ef4c0b
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/70FictWildeO_Dorian_18_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Chapter 18 The next day he did not leave the house , and , indeed , spent most of the time in his own room , sick with a wild terror of dying , and yet indifferent to life itself . The consciousness of being hunted , snared , tracked down , had begun to dominate him . If the tapestry did but tremble in the wind , he shook . The dead leaves that were blown against the leaded panes seemed to him like his own wasted resolutions and wild regrets . When he closed his eyes , he saw again the sailor 's face peering through the mist-stained glass , and horror seemed once more to lay its hand upon his heart . But perhaps it had been only his fancy that had called vengeance out of the night and set the hideous shapes of punishment before him . Actual life was chaos , but there was something terribly logical in the imagination . It was the imagination that set remorse to dog the feet of sin . It was the imagination that made each crime bear its misshapen brood . In the common world of fact
  the wicked were not punished , nor the good rewarded . Success was given to the strong , failure thrust upon the weak . That was all . Besides , had any stranger been prowling round the house , he would have been seen by the servants or the keepers . Had any foot-marks been found on the flower-beds , the gardeners would have reported it . Yes , it had been merely fancy . Sibyl Vane 's brother had not come back to kill him . He had sailed away in his ship to founder in some winter sea . From him , at any rate , he was safe . Why , the man did not know who he was , could not know who he was . The mask of youth had saved him . And yet if it had been merely an illusion , how terrible it was to think that conscience could raise such fearful phantoms , and give them visible form , and make them move before one ! What sort of life would his be if , day and night , shadows of his crime were to peer at him from silent corners , to mock him from secret places , to whisper in his ear as he sa
 t at the feast , to wake him with icy fingers as he lay asleep ! As the thought crept through his brain , he grew pale with terror , and the air seemed to him to have become suddenly colder . Oh ! in what a wild hour of madness he had killed his friend ! How ghastly the mere memory of the scene ! He saw it all again . Each hideous detail came back to him with added horror . Out of the black cave of time , terrible and swathed in scarlet , rose the image of his sin . When Lord Henry came in at six o'clock , he found him crying as one whose heart will break . It was not till the third day that he ventured to go out . There was something in the clear , pine-scented air of that winter morning that seemed to bring him back his joyousness and his ardour for life . But it was not merely the physical conditions of environment that had caused the change . His own nature had revolted against the excess of anguish that had sought to maim and mar the perfection of its calm . With subtle and fin
 ely wrought temperaments it is always so . Their strong passions must either bruise or bend . They either slay the man , or themselves die . Shallow sorrows and shallow loves live on . The loves and sorrows that are great are destroyed by their own plenitude . Besides , he had convinced himself that he had been the victim of a terror-stricken imagination , and looked back now on his fears with something of pity and not a little of contempt . After breakfast , he walked with the Duchess for an hour in the garden and then drove across the park to join the shooting-party . The crisp frost lay like salt upon the grass . The sky was an inverted cup of blue metal . A thin film of ice bordered the flat , reed-grown lake . At the corner of the pine-wood he caught sight of Sir Geoffrey Clouston , the Duchess 's brother , jerking two spent cartridges out of his gun . He jumped from the cart , and having told the groom to take the mare home , made his way towards his guest through the withered
  bracken and rough undergrowth . " Have you had good sport , Geoffrey ? " he asked . " Not very good , Dorian . I think most of the birds have gone to the open . I dare say it will be better after lunch , when we get to new ground . " Dorian strolled along by his side . The keen aromatic air , the brown and red lights that glimmered in the wood , the hoarse cries of the beaters ringing out from time to time , and the sharp snaps of the guns that followed , fascinated him and filled him with a sense of delightful freedom . He was dominated by the carelessness of happiness , by the high indifference of joy . Suddenly from a lumpy tussock of old grass some twenty yards in front of them , with black-tipped ears erect and long hinder limbs throwing it forward , started a hare . It bolted for a thicket of alders . Sir Geoffrey put his gun to his shoulder , but there was something in the animal 's grace of movement that strangely charmed Dorian Gray , and he cried out at once , " Do n't sh
 oot it , Geoffrey . Let it live . " " What nonsense , Dorian ! " laughed his companion , and as the hare bounded into the thicket , he fired . There were two cries heard , the cry of a hare in pain , which is dreadful , the cry of a man in agony , which is worse . " Good heavens ! I have hit a beater ! " exclaimed Sir Geoffrey . " What an ass the man was to get in front of the guns ! Stop shooting there ! " he called out at the top of his voice . " A man is hurt . " The head-keeper came running up with a stick in his hand . " Where , sir ? Where is he ? " he shouted . At the same time , the firing ceased along the line . " Here , " answered Sir Geoffrey angrily , hurrying towards the thicket . " Why on earth do n't you keep your men back ? Spoiled my shooting for the day . " Dorian watched them as they plunged into the alder-clump , brushing the lithe swinging branches aside . In a few moments they emerged , dragging a body after them into the sunlight . He turned away in horror . I
 t seemed to him that misfortune followed wherever he went . He heard Sir Geoffrey ask if the man was really dead , and the affirmative answer of the keeper . The wood seemed to him to have become suddenly alive with faces . There was the trampling of myriad feet and the low buzz of voices . A great copper-breasted pheasant came beating through the boughs overhead . After a few moments \u2013 that were to him , in his perturbed state , like endless hours of pain \u2013 he felt a hand laid on his shoulder . He started and looked round . " Dorian , " said Lord Henry , " I had better tell them that the shooting is stopped for to-day . It would not look well to go on . " " I wish it were stopped for ever , Harry , " he answered bitterly . " The whole thing is hideous and cruel . Is the man ... ? " He could not finish the sentence . " I am afraid so , " rejoined Lord Henry . " He got the whole charge of shot in his chest . He must have died almost instantaneously . Come ; let us go home . " The
 y walked side by side in the direction of the avenue for nearly fifty yards without speaking . Then Dorian looked at Lord Henry and said , with a heavy sigh , " It is a bad omen , Harry , a very bad omen . " " What is ? " asked Lord Henry . " Oh ! this accident , I suppose . My dear fellow , it ca n't be helped . It was the man 's own fault . Why did he get in front of the guns ? Besides , it is nothing to us . It is rather awkward for Geoffrey , of course . It does not do to pepper beaters . It makes people think that one is a wild shot . And Geoffrey is not ; he shoots very straight . But there is no use talking about the matter . " Dorian shook his head . " It is a bad omen , Harry . I feel as if something horrible were going to happen to some of us . To myself , perhaps , " he added , passing his hand over his eyes , with a gesture of pain . The elder man laughed . " The only horrible thing in the world is ennui , Dorian . That is the one sin for which there is no forgiveness . 
 But we are not likely to suffer from it unless these fellows keep chattering about this thing at dinner . I must tell them that the subject is to be tabooed . As for omens , there is no such thing as an omen . Destiny does not send us heralds . She is too wise or too cruel for that . Besides , what on earth could happen to you , Dorian ? You have everything in the world that a man can want . There is no one who would not be delighted to change places with you . " " There is no one with whom I would not change places , Harry . Do n't laugh like that . I am telling you the truth . The wretched peasant who has just died is better off than I am . I have no terror of death . It is the coming of death that terrifies me . Its monstrous wings seem to wheel in the leaden air around me . Good heavens ! do n't you see a man moving behind the trees there , watching me , waiting for me ? " Lord Henry looked in the direction in which the trembling gloved hand was pointing . " Yes , " he said , sm
 iling , " I see the gardener waiting for you . I suppose he wants to ask you what flowers you wish to have on the table to-night . How absurdly nervous you are , my dear fellow ! You must come and see my doctor , when we get back to town . " Dorian heaved a sigh of relief as he saw the gardener approaching . The man touched his hat , glanced for a moment at Lord Henry in a hesitating manner , and then produced a letter , which he handed to his master . " Her Grace told me to wait for an answer , " he murmured . Dorian put the letter into his pocket . " Tell her Grace that I am coming in , " he said , coldly . The man turned round and went rapidly in the direction of the house . " How fond women are of doing dangerous things ! " laughed Lord Henry . " It is one of the qualities in them that I admire most . A woman will flirt with anybody in the world as long as other people are looking on . " " How fond you are of saying dangerous things , Harry ! In the present instance , you are qu
 ite astray . I like the Duchess very much , but I do n't love her . " " And the Duchess loves you very much , but she likes you less , so you are excellently matched . " " You are talking scandal , Harry , and there is never any basis for scandal . " " The basis of every scandal is an immoral certainty , " said Lord Henry , lighting a cigarette . " You would sacrifice anybody , Harry , for the sake of an epigram . " " The world goes to the altar of its own accord , " was the answer . " I wish I could love , " cried Dorian Gray with a deep note of pathos in his voice . " But I seem to have lost the passion and forgotten the desire . I am too much concentrated on myself . My own personality has become a burden to me . I want to escape , to go away , to forget . It was silly of me to come down here at all . I think I shall send a wire to Harvey to have the yacht got ready . On a yacht one is safe . " " Safe from what , Dorian ? You are in some trouble . Why not tell me what it is ? You
  know I would help you . " " I ca n't tell you , Harry , " he answered sadly . " And I dare say it is only a fancy of mine . This unfortunate accident has upset me . I have a horrible presentiment that something of the kind may happen to me . " " What nonsense ! " " I hope it is , but I ca n't help feeling it . Ah ! here is the Duchess , looking like Artemis in a tailor-made gown . You see we have come back , Duchess . " " I have heard all about it , Mr. Gray , " she answered . " Poor Geoffrey is terribly upset . And it seems that you asked him not to shoot the hare . How curious ! " " Yes , it was very curious . I do n't know what made me say it . Some whim , I suppose . It looked the loveliest of little live things . But I am sorry they told you about the man . It is a hideous subject . " " It is an annoying subject , " broke in Lord Henry . " It has no psychological value at all . Now if Geoffrey had done the thing on purpose , how interesting he would be ! I should like to know 
 some one who had committed a real murder . " " How horrid of you , Harry ! " cried the Duchess . " Is n't it , Mr. Gray ? Harry , Mr. Gray is ill again . He is going to faint . " Dorian drew himself up with an effort and smiled . " It is nothing , Duchess , " he murmured ; " my nerves are dreadfully out of order . That is all . I am afraid I walked too far this morning . I did n't hear what Harry said . Was it very bad ? You must tell me some other time . I think I must go and lie down . You will excuse me , wo n't you ? " They had reached the great flight of steps that led from the conservatory on to the terrace . As the glass door closed behind Dorian , Lord Henry turned and looked at the Duchess with his slumberous eyes . " Are you very much in love with him ? " he asked . She did not answer for some time , but stood gazing at the landscape . " I wish I knew , " she said at last . He shook his head . " Knowledge would be fatal . It is the uncertainty that charms one . A mist make
 s things wonderful . " " One may lose one 's way . " " All ways end at the same point , my dear Gladys . " " What is that ? " " Disillusion . " " It was my d�but in life , " she sighed . " It came to you crowned . " " I am tired of strawberry leaves . " " They become you . " " Only in public . " " You would miss them , " said Lord Henry . " I will not part with a petal . " " Monmouth has ears . " " Old age is dull of hearing . " " Has he never been jealous ? " " I wish he had been . " He glanced about as if in search of something . " What are you looking for ? " she inquired . " The button from your foil , " he answered . " You have dropped it . " She laughed . " I have still the mask . " " It makes your eyes lovelier , " was his reply . She laughed again . Her teeth showed like white seeds in a scarlet fruit . Upstairs , in his own room , Dorian Gray was lying on a sofa , with terror in every tingling fibre of his body . Life had suddenly become too hideous a burden for him to bea
 r . The dreadful death of the unlucky beater , shot in the thicket like a wild animal , had seemed to him to pre-figure death for himself also . He had nearly swooned at what Lord Henry had said in a chance mood of cynical jesting . At five o'clock he rang his bell for his servant and gave him orders to pack his things for the night-express to town , and to have the brougham at the door by eight-thirty . He was determined not to sleep another night at Selby Royal . It was an ill-omened place . Death walked there in the sunlight . The grass of the forest had been spotted with blood . Then he wrote a note to Lord Henry , telling him that he was going up to town to consult his doctor and asking him to entertain his guests in his absence . As he was putting it into the envelope , a knock came to the door , and his valet informed him that the head-keeper wished to see him . He frowned and bit his lip . " Send him in , " he muttered , after some moments ' hesitation . As soon as the man e
 ntered , Dorian pulled his chequebook out of a drawer and spread it out before him . " I suppose you have come about the unfortunate accident of this morning , Thornton ? " he said , taking up a pen . " Yes , sir , " answered the gamekeeper . " Was the poor fellow married ? Had he any people dependent on him ? " asked Dorian , looking bored . " If so , I should not like them to be left in want , and will send them any sum of money you may think necessary . " " We do n't know who he is , sir . That is what I took the liberty of coming to you about . " " Do n't know who he is ? " said Dorian , listlessly . " What do you mean ? Was n't he one of your men ? " " No , sir . Never saw him before . Seems like a sailor , sir . " The pen dropped from Dorian Gray 's hand , and he felt as if his heart had suddenly stopped beating . " A sailor ? " he cried out . " Did you say a sailor ? " " Yes , sir . He looks as if he had been a sort of sailor ; tattooed on both arms , and that kind of thing .
  " " Was there anything found on him ? " said Dorian , leaning forward and looking at the man with startled eyes . " Anything that would tell his name ? " " Some money , sir \u2013 not much , and a six-shooter . There was no name of any kind . A decent-looking man , sir , but rough-like . A sort of sailor we think . " Dorian started to his feet . A terrible hope fluttered past him . He clutched at it madly . " Where is the body ? " he exclaimed . " Quick ! I must see it at once . " " It is in an empty stable in the Home Farm , sir . The folk do n't like to have that sort of thing in their houses . They say a corpse brings bad luck . " " The Home Farm ! Go there at once and meet me . Tell one of the grooms to bring my horse round . No. Never mind . I 'll go to the stables myself . It will save time . " In less than a quarter of an hour , Dorian Gray was galloping down the long avenue as hard as he could go . The trees seemed to sweep past him in spectral procession , and wild shadows to
  fling themselves across his path . Once the mare swerved at a white gate-post and nearly threw him . He lashed her across the neck with his crop . She cleft the dusky air like an arrow . The stones flew from her hoofs . At last he reached the Home Farm . Two men were loitering in the yard . He leaped from the saddle and threw the reins to one of them . In the farthest stable a light was glimmering . Something seemed to tell him that the body was there , and he hurried to the door and put his hand upon the latch . There he paused for a moment , feeling that he was on the brink of a discovery that would either make or mar his life . Then he thrust the door open and entered . On a heap of sacking in the far corner was lying the dead body of a man dressed in a coarse shirt and a pair of blue trousers . A spotted handkerchief had been placed over the face . A coarse candle , stuck in a bottle , sputtered beside it . Dorian Gray shuddered . He felt that his could not be the hand to take 
 the handkerchief away , and called out to one of the farm-servants to come to him . " Take that thing off the face . I wish to see it , " he said , clutching at the door-post for support . When the farm-servant had done so , he stepped forward . A cry of joy broke from his lips . The man who had been shot in the thicket was James Vane . He stood there for some minutes looking at the dead body . As he rode home , his eyes were full of tears , for he knew he was safe . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/71FictWildeO_Dorian_8_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/71FictWildeO_Dorian_8_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/71FictWildeO_Dorian_8_EN.txt.txt
new file mode 100644
index 0000000..2ee7b94
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/71FictWildeO_Dorian_8_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Chapter 8 It was long past noon when he awoke . His valet had crept several times on tiptoe into the room to see if he was stirring , and had wondered what made his young master sleep so late . Finally his bell sounded , and Victor came softly in with a cup of tea , and a pile of letters , on a small tray of old Sevres china , and drew back the olive-satin curtains , with their shimmering blue lining , that hung in front of the three tall windows . " Monsieur has slept well this morning , " he said , smiling . " What o'clock is it , Victor ? " asked Dorian Gray drowsily . " One hour and a quarter , Monsieur . " How late it was ! He sat up , and having sipped some tea , turned over his letters . One of them was from Lord Henry , and had been brought by hand that morning . He hesitated for a moment , and then put it aside . The others he opened listlessly . They contained the usual collection of cards , invitations to dinner , tickets for private views , programmes of charity conc
 erts , and the like that are showered on fashionable young men every morning during the season . There was a rather heavy bill for a chased silver Louis-Quinze toilet-set that he had not yet had the courage to send on to his guardians , who were extremely old-fashioned people and did not realise that we live in an age when unnecessary things are our only necessities ; and there were several very courteously worded communications from Jermyn Street money-lenders offering to advance any sum of money at a moment 's notice and at the most reasonable rates of interest . After about ten minutes he got up , and throwing on an elaborate dressing-gown of silk-embroidered cashmere wool , passed into the onyx-paved bathroom . The cool water refreshed him after his long sleep . He seemed to have forgotten all that he had gone through . A dim sense of having taken part in some strange tragedy came to him once or twice , but there was the unreality of a dream about it . As soon as he was dressed 
 , he went into the library and sat down to a light French breakfast that had been laid out for him on a small round table close to the open window . It was an exquisite day . The warm air seemed laden with spices . A bee flew in and buzzed round the blue-dragon bowl that , filled with sulphur-yellow roses , stood before him . He felt perfectly happy . Suddenly his eye fell on the screen that he had placed in front of the portrait , and he started . " Too cold for Monsieur ? " asked his valet , putting an omelette on the table . " I shut the window ? " Dorian shook his head . " I am not cold , " he murmured . Was it all true ? Had the portrait really changed ? Or had it been simply his own imagination that had made him see a look of evil where there had been a look of joy ? Surely a painted canvas could not alter ? The thing was absurd . It would serve as a tale to tell Basil some day . It would make him smile . And yet , how vivid was his recollection of the whole thing ! First in t
 he dim twilight , and then in the bright dawn , he had seen the touch of cruelty round the warped lips . He almost dreaded his valet leaving the room . He knew that when he was alone he would have to examine the portrait . He was afraid of certainty . When the coffee and cigarettes had been brought and the man turned to go , he felt a wild desire to tell him to remain . As the door was closing behind him , he called him back . The man stood waiting for his orders . Dorian looked at him for a moment . " I am not at home to any one , Victor , " he said with a sigh . The man bowed and retired . Then he rose from the table , lit a cigarette , and flung himself down on a luxuriously cushioned couch that stood facing the screen . The screen was an old one , of gilt Spanish leather , stamped and wrought with a rather florid Louis-Quatorze pattern . He scanned it curiously , wondering if ever before it had concealed the secret of a man 's life . Should he move it aside , after all ? Why not
  let it stay there ? What was the use of knowing . ? If the thing was true , it was terrible . If it was not true , why trouble about it ? But what if , by some fate or deadlier chance , eyes other than his spied behind and saw the horrible change ? What should he do if Basil Hallward came and asked to look at his own picture ? Basil would be sure to do that . No ; the thing had to be examined , and at once . Anything would be better than this dreadful state of doubt . He got up and locked both doors . At least he would be alone when he looked upon the mask of his shame . Then he drew the screen aside and saw himself face to face . It was perfectly true . The portrait had altered . As he often remembered afterwards , and always with no small wonder , he found himself at first gazing at the portrait with a feeling of almost scientific interest . That such a change should have taken place was incredible to him . And yet it was a fact . Was there some subtle affinity between the chemic
 al atoms that shaped themselves into form and colour on the canvas and the soul that was within him ? Could it be that what that soul thought , they realised ? \u2013 that what it dreamed , they made true ? Or was there some other , more terrible reason ? He shuddered , and felt afraid , and , going back to the couch , lay there , gazing at the picture in sickened horror . One thing , however , he felt that it had done for him . It had made him conscious how unjust , how cruel , he had been to Sibyl Vane . It was not too late to make reparation for that . She could still be his wife . His unreal and selfish love would yield to some higher influence , would be transformed into some nobler passion , and the portrait that Basil Hallward had painted of him would be a guide to him through life , would be to him what holiness is to some , and conscience to others , and the fear of God to us all . There were opiates for remorse , drugs that could lull the moral sense to sleep . But here was a
  visible symbol of the degradation of sin . Here was an ever-present sign of the ruin men brought upon their souls . Three o'clock struck , and four , and the half-hour rang its double chime , but Dorian Gray did not stir . He was trying to gather up the scarlet threads of life and to weave them into a pattern ; to find his way through the sanguine labyrinth of passion through which he was wandering . He did not know what to do , or what to think . Finally , he went over to the table and wrote a passionate letter to the girl he had loved , imploring her forgiveness and accusing himself of madness . He covered page after page with wild words of sorrow and wilder words of pain . There is a luxury in self-reproach . When we blame ourselves , we feel that no one else has a right to blame us . It is the confession , not the priest , that gives us absolution . When Dorian had finished the letter , he felt that he had been forgiven . Suddenly there came a knock to the door , and he heard L
 ord Henry 's voice outside . " My dear boy , I must see you . Let me in at once . I ca n't bear your shutting yourself up like this . " He made no answer at first , but remained quite still . The knocking still continued and grew louder . Yes , it was better to let Lord Henry in , and to explain to him the new life he was going to lead , to quarrel with him if it became necessary to quarrel , to part if parting was inevitable . He jumped up , drew the screen hastily across the picture , and unlocked the door . " I am so sorry for it all , Dorian , " said Lord Henry as he entered . " But you must not think too much about it . " " Do you mean about Sibyl Vane ? " asked the lad . " Yes , of course , " answered Lord Henry , sinking into a chair and slowly pulling off his yellow gloves . " It is dreadful , from one point of view , but it was not your fault . Tell me , did you go behind and see her , after the play was over ? " " Yes . " " I felt sure you had . Did you make a scene with h
 er ? " " I was brutal , Harry \u2013 perfectly brutal . But it is all right now . I am not sorry for anything that has happened . It has taught me to know myself better . " " Ah , Dorian , I am so glad you take it in that way ! I was afraid I would find you plunged in remorse and tearing that nice curly hair of yours . " " I have got through all that , " said Dorian , shaking his head and smiling . " I am perfectly happy now . I know what conscience is , to begin with . It is not what you told me it was . It is the divinest thing in us . Do n't sneer at it , Harry , any more \u2013 at least not before me . I want to be good . I ca n't bear the idea of my soul being hideous . " " A very charming artistic basis for ethics , Dorian ! I congratulate you on it . But how are you going to begin ? " " By marrying Sibyl Vane . " " Marrying Sibyl Vane ! " cried Lord Henry , standing up and looking at him in perplexed amazement . " But , my dear Dorian \u2013 " " Yes , Harry , I know what you are going
  to say . Something dreadful about marriage . Do n't say it . Do n't ever say things of that kind to me again . Two days ago I asked Sibyl to marry me . I am not going to break my word to her . She is to be my wife . " " Your wife ! Dorian ! ... Did n't you get my letter ? I wrote to you this morning , and sent the note down by my own man . " " Your letter ? Oh , yes , I remember . I have not read it yet , Harry . I was afraid there might be something in it that I would n't like . You cut life to pieces with your epigrams . " " You know nothing then ? " " What do you mean ? " Lord Henry walked across the room , and sitting down by Dorian Gray , took both his hands in his own and held them tightly . " Dorian , " he said , " my letter \u2013 do n't be frightened \u2013 was to tell you that Sibyl Vane is dead . " A cry of pain broke from the lad 's lips , and he leaped to his feet , tearing his hands away from Lord Henry 's grasp . " Dead ! Sibyl dead ! It is not true ! It is a horrible lie 
 ! How dare you say it ? " " It is quite true , Dorian , " said Lord Henry , gravely . " It is in all the morning papers . I wrote down to you to ask you not to see any one till I came . There will have to be an inquest , of course , and you must not be mixed up in it . Things like that make a man fashionable in Paris . But in London people are so prejudiced . Here , one should never make one 's d�but with a scandal . One should reserve that to give an interest to one 's old age . I suppose they do n't know your name at the theatre ? If they do n't , it is all right . Did any one see you going round to her room ? That is an important point . " Dorian did not answer for a few moments . He was dazed with horror . Finally he stammered , in a stifled voice , " Harry , did you say an inquest ? What did you mean by that ? Did Sibyl \u2013 ? Oh , Harry , I ca n't bear it ! But be quick . Tell me everything at once . " " I have no doubt it was not an accident , Dorian , though it must be put i
 n that way to the public . It seems that as she was leaving the theatre with her mother , about half-past twelve or so , she said she had forgotten something upstairs . They waited some time for her , but she did not come down again . They ultimately found her lying dead on the floor of her dressing-room . She had swallowed something by mistake , some dreadful thing they use at theatres . I do n't know what it was , but it had either prussic acid or white lead in it . I should fancy it was prussic acid , as she seems to have died instantaneously . " " Harry , Harry , it is terrible ! " cried the lad . " Yes ; it is very tragic , of course , but you must not get yourself mixed up in it . I see by the Standard that she was seventeen . I should have thought she was almost younger than that . She looked such a child , and seemed to know so little about acting . Dorian , you must n't let this thing get on your nerves . You must come and dine with me , and afterwards we will look in at th
 e opera . It is a Patti night , and everybody will be there . You can come to my sister 's box . She has got some smart women with her . " " So I have murdered Sibyl Vane , " said Dorian Gray , half to himself , " murdered her as surely as if I had cut her little throat with a knife . Yet the roses are not less lovely for all that . The birds sing just as happily in my garden . And to-night I am to dine with you , and then go on to the opera , and sup somewhere , I suppose , afterwards . How extraordinarily dramatic life is ! If I had read all this in a book , Harry , I think I would have wept over it . Somehow , now that it has happened actually , and to me , it seems far too wonderful for tears . Here is the first passionate love-letter I have ever written in my life . Strange , that my first passionate love-letter should have been addressed to a dead girl . Can they feel , I wonder , those white silent people we call the dead ? Sibyl ! Can she feel , or know , or listen ? Oh , Ha
 rry , how I loved her once ! It seems years ago to me now . She was everything to me . Then came that dreadful night \u2013 was it really only last night ? \u2013 when she played so badly , and my heart almost broke . She explained it all to me . It was terribly pathetic . But I was not moved a bit . I thought her shallow . Suddenly something happened that made me afraid . I ca n't tell you what it was , but it was terrible . I said I would go back to her . I felt I had done wrong . And now she is dead . My God ! My God ! Harry , what shall I do ? You do n't know the danger I am in , and there is nothing to keep me straight . She would have done that for me . She had no right to kill herself . It was selfish of her . " " My dear Dorian , " answered Lord Henry , taking a cigarette from his case and producing a gold-latten matchbox , " the only way a woman can ever reform a man is by boring him so completely that he loses all possible interest in life . If you had married this girl , you wo
 uld have been wretched . Of course , you would have treated her kindly . One can always be kind to people about whom one cares nothing . But she would have soon found out that you were absolutely indifferent to her . And when a woman finds that out about her husband , she either becomes dreadfully dowdy , or wears very smart bonnets that some other woman 's husband has to pay for . I say nothing about the social mistake , which would have been abject \u2013 which , of course , I would not have allowed \u2013 but I assure you that in any case the whole thing would have been an absolute failure . " " I suppose it would , " muttered the lad , walking up and down the room and looking horribly pale . " But I thought it was my duty . It is not my fault that this terrible tragedy has prevented my doing what was right . I remember your saying once that there is a fatality about good resolutions \u2013 that they are always made too late . Mine certainly were . " " Good resolutions are useless attempt
 s to interfere with scientific laws . Their origin is pure vanity . Their result is absolutely nil . They give us , now and then , some of those luxurious sterile emotions that have a certain charm for the weak . That is all that can be said for them . They are simply cheques that men draw on a bank where they have no account . " " Harry , " cried Dorian Gray , coming over and sitting down beside him , " why is it that I cannot feel this tragedy as much as I want to ? I do n't think I am heartless . Do you ? " " You have done too many foolish things during the last fortnight to be entitled to give yourself that name , Dorian , " answered Lord Henry with his sweet melancholy smile . The lad frowned . " I do n't like that explanation , Harry , " he rejoined , " but I am glad you do n't think I am heartless . I am nothing of the kind . I know I am not . And yet I must admit that this thing that has happened does not affect me as it should . It seems to me to be simply like a wonderful 
 ending to a wonderful play . It has all the terrible beauty of a Greek tragedy , a tragedy in which I took a great part , but by which I have not been wounded . " " It is an interesting question , " said Lord Henry , who found an exquisite pleasure in playing on the lad 's unconscious egotism \u2013 " an extremely interesting question . I fancy that the true explanation is this : It often happens that the real tragedies of life occur in such an inartistic manner that they hurt us by their crude violence , their absolute incoherence , their absurd want of meaning , their entire lack of style . They affect us just as vulgarity affects us . They give us an impression of sheer brute force , and we revolt against that . Sometimes , however , a tragedy that possesses artistic elements of beauty crosses our lives . If these elements of beauty are real , the whole thing simply appeals to our sense of dramatic effect . Suddenly we find that we are no longer the actors , but the spectators of th
 e play . Or rather we are both . We watch ourselves , and the mere wonder of the spectacle enthralls us . In the present case , what is it that has really happened ? Some one has killed herself for love of you . I wish that I had ever had such an experience . It would have made me in love with love for the rest of my life . The people who have adored me \u2013 there have not been very many , but there have been some \u2013 have always insisted on living on , long after I had ceased to care for them , or they to care for me . They have become stout and tedious , and when I meet them , they go in at once for reminiscences . That awful memory of woman ! What a fearful thing it is ! And what an utter intellectual stagnation it reveals ! One should absorb the colour of life , but one should never remember its details . Details are always vulgar . " " I must sow poppies in my garden , " sighed Dorian . " There is no necessity , " rejoined his companion . " Life has always poppies in her hands .
  Of course , now and then things linger . I once wore nothing but violets all through one season , as a form of artistic mourning for a romance that would not die . Ultimately , however , it did die . I forget what killed it . I think it was her proposing to sacrifice the whole world for me . That is always a dreadful moment . It fills one with the terror of eternity . Well \u2013 would you believe it ? \u2013 a week ago , at Lady Hampshire 's , I found myself seated at dinner next the lady in question , and she insisted on going over the whole thing again , and digging up the past , and raking up the future . I had buried my romance in a bed of asphodel . She dragged it out again and assured me that I had spoiled her life . I am bound to state that she ate an enormous dinner , so I did not feel any anxiety . But what a lack of taste she showed ! The one charm of the past is that it is the past . But women never know when the curtain has fallen . They always want a sixth act , and as soon
  as the interest of the play is entirely over , they propose to continue it . If they were allowed their own way , every comedy would have a tragic ending , and every tragedy would culminate in a farce . They are charmingly artificial , but they have no sense of art . You are more fortunate than I am . I assure you , Dorian , that not one of the women I have known would have done for me what Sibyl Vane did for you . Ordinary women always console themselves . Some of them do it by going in for sentimental colours . Never trust a woman who wears mauve , whatever her age may be , or a woman over thirty-five who is fond of pink ribbons . It always means that they have a history . Others find a great consolation in suddenly discovering the good qualities of their husbands . They flaunt their conjugal felicity in one 's face , as if it were the most fascinating of sins . Religion consoles some . Its mysteries have all the charm of a flirtation , a woman once told me , and I can quite unde
 rstand it . Besides , nothing makes one so vain as being told that one is a sinner . Conscience makes egotists of us all . Yes ; there is really no end to the consolations that women find in modern life . Indeed , I have not mentioned the most important one . " " What is that , Harry ? " said the lad listlessly . " Oh , the obvious consolation . Taking some one else 's admirer when one loses one 's own . In good society that always whitewashes a woman . But really , Dorian , how different Sibyl Vane must have been from all the women one meets ! There is something to me quite beautiful about her death . I am glad I am living in a century when such wonders happen . They make one believe in the reality of the things we all play with , such as romance , passion , and love . " " I was terribly cruel to her . You forget that . " " I am afraid that women appreciate cruelty , downright cruelty , more than anything else . They have wonderfully primitive instincts . We have emancipated them ,
  but they remain slaves looking for their masters , all the same . They love being dominated . I am sure you were splendid . I have never seen you really and absolutely angry , but I can fancy how delightful you looked . And , after all , you said something to me the day before yesterday that seemed to me at the time to be merely fanciful , but that I see now was absolutely true , and it holds the key to everything . " " What was that , Harry ? " " You said to me that Sibyl Vane represented to you all the heroines of romance \u2013 that she was Desdemona one night , and Ophelia the other ; that if she died as Juliet , she came to life as Imogen . " " She will never come to life again now , " muttered the lad , burying his face in his hands . " No , she will never come to life . She has played her last part . But you must think of that lonely death in the tawdry dressing-room simply as a strange lurid fragment from some Jacobean tragedy , as a wonderful scene from Webster , or Ford , or
  Cyril Tourneur . The girl never really lived , and so she has never really died . To you at least she was always a dream , a phantom that flitted through Shakespeare 's plays and left them lovelier for its presence , a reed through which Shakespeare 's music sounded richer and more full of joy . The moment she touched actual life , she marred it , and it marred her , and so she passed away . Mourn for Ophelia , if you like . Put ashes on your head because Cordelia was strangled . Cry out against Heaven because the daughter of Brabantio died . But do n't waste your tears over Sibyl Vane . She was less real than they are . " There was a silence . The evening darkened in the room . Noiselessly , and with silver feet , the shadows crept in from the garden . The colours faded wearily out of things . After some time Dorian Gray looked up . " You have explained me to myself , Harry , " he murmured with something of a sigh of relief . " I felt all that you have said , but somehow I was afr
 aid of it , and I could not express it to myself . How well you know me ! But we will not talk again of what has happened . It has been a marvellous experience . That is all . I wonder if life has still in store for me anything as marvellous . " " Life has everything in store for you , Dorian . There is nothing that you , with your extraordinary good looks , will not be able to do . " " But suppose , Harry , I became haggard , and old , and wrinkled ? What then ? " " Ah , then , " said Lord Henry , rising to go \u2013 " then , my dear Dorian , you would have to fight for your victories . As it is , they are brought to you . No , you must keep your good looks . We live in an age that reads too much to be wise , and that thinks too much to be beautiful . We cannot spare you . And now you had better dress and drive down to the club . We are rather late , as it is . " " I think I shall join you at the opera , Harry . I feel too tired to eat anything . What is the number of your sister 's b
 ox ? " " Twenty-seven , I believe . It is on the grand tier . You will see her name on the door . But I am sorry you wo n't come and dine . " " I do n't feel up to it , " said Dorian listlessly . " But I am awfully obliged to you for all that you have said to me . You are certainly my best friend . No one has ever understood me as you have . " " We are only at the beginning of our friendship , Dorian , " answered Lord Henry , shaking him by the hand . " Good-bye . I shall see you before nine-thirty , I hope . Remember , Patti is singing . " As he closed the door behind him , Dorian Gray touched the bell , and in a few minutes Victor appeared with the lamps and drew the blinds down . He waited impatiently for him to go . The man seemed to take an interminable time over everything . As soon as he had left , he rushed to the screen and drew it back . No ; there was no further change in the picture . It had received the news of Sibyl Vane 's death before he had known of it himself . It 
 was conscious of the events of life as they occurred . The vicious cruelty that marred the fine lines of the mouth had , no doubt , appeared at the very moment that the girl had drunk the poison , whatever it was . Or was it indifferent to results ? Did it merely take cognizance of what passed within the soul ? He wondered , and hoped that some day he would see the change taking place before his very eyes , shuddering as he hoped it . Poor Sibyl ! What a romance it had all been ! She had often mimicked death on the stage . Then Death himself had touched her and taken her with him . How had she played that dreadful last scene ? Had she cursed him , as she died ? No ; she had died for love of him , and love would always be a sacrament to him now . She had atoned for everything by the sacrifice she had made of her life . He would not think any more of what she had made him go through , on that horrible night at the theatre . When he thought of her , it would be as a wonderful tragic fi
 gure sent on to the world 's stage to show the supreme reality of love . A wonderful tragic figure ? Tears came to his eyes as he remembered her childlike look , and winsome fanciful ways , and shy tremulous grace . He brushed them away hastily and looked again at the picture . He felt that the time had really come for making his choice . Or had his choice already been made ? Yes , life had decided that for him \u2013 life , and his own infinite curiosity about life . Eternal youth , infinite passion , pleasures subtle and secret , wild joys and wilder sins \u2013 he was to have all these things . The portrait was to bear the burden of his shame : that was all . A feeling of pain crept over him as he thought of the desecration that was in store for the fair face on the canvas . Once , in boyish mockery of Narcissus , he had kissed , or feigned to kiss , those painted lips that now smiled so cruelly at him . Morning after morning he had sat before the portrait wondering at its beauty , alm
 ost enamoured of it , as it seemed to him at times . Was it to alter now with every mood to which he yielded ? Was it to become a monstrous and loathsome thing , to be hidden away in a locked room , to be shut out from the sunlight that had so often touched to brighter gold the waving wonder of its hair ? The pity of it ! the pity of it ! For a moment , he thought of praying that the horrible sympathy that existed between him and the picture might cease . It had changed in answer to a prayer ; perhaps in answer to a prayer it might remain unchanged . And yet , who , that knew anything about life , would surrender the chance of remaining always young , however fantastic that chance might be , or with what fateful consequences it might be fraught ? Besides , was it really under his control ? Had it indeed been prayer that had produced the substitution ? Might there not be some curious scientific reason for it all ? If thought could exercise its influence upon a living organism , might
  not thought exercise an influence upon dead and inorganic things ? Nay , without thought or conscious desire , might not things external to ourselves vibrate in unison with our moods and passions , atom calling to atom in secret love or strange affinity ? But the reason was of no importance . He would never again tempt by a prayer any terrible power . If the picture was to alter , it was to alter . That was all . Why inquire too closely into it ? For there would be a real pleasure in watching it . He would be able to follow his mind into its secret places . This portrait would be to him the most magical of mirrors . As it had revealed to him his own body , so it would reveal to him his own soul . And when winter came upon it , he would still be standing where spring trembles on the verge of summer . When the blood crept from its face , and left behind a pallid mask of chalk with leaden eyes , he would keep the glamour of boyhood . Not one blossom of his loveliness would ever fade .
  Not one pulse of his life would ever weaken . Like the gods of the Greeks , he would be strong , and fleet , and joyous . What did it matter what happened to the coloured image on the canvas ? He would be safe . That was everything . He drew the screen back into its former place in front of the picture , smiling as he did so , and passed into his bedroom , where his valet was already waiting for him . An hour later he was at the opera , and Lord Henry was leaning over his chair . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/72FictZolaE_Germinal_II4_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/72FictZolaE_Germinal_II4_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/72FictZolaE_Germinal_II4_EN.txt.txt
new file mode 100644
index 0000000..6d9ccbb
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/72FictZolaE_Germinal_II4_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Part 2 Chapter 4 WHEN Maheu came in after having left �tienne at Rasseneur 's , he found Catherine , Zacharie , and Jeanlin seated at the table finishing their soup . On returning from the pit they were always so hungry that they ate in their damp clothes , without even cleaning themselves ; and no one was waited for , the table was laid from morning to night ; there was always someone there swallowing his portion , according to the chances of work . As he entered the door Maheu saw the provisions . He said nothing , but his uneasy face lighted up . All the morning the emptiness of the cupboard , the thought of the house without coffee and without butter , had been troubling him ; the recollection came to him painfully while he was hammering at the seam , stifled at the bottom of the cutting . What would his wife do , and what would become of them if she were to return with empty hands ? And now , here was everything ! She would tell him about it later on . He laughed with sati
 sfaction . Catherine and Jeanlin had risen , and were taking their coffee standing ; while Zacharie , not filled with the soup , cut himself a large slice of bread and covered it with butter . Although he saw the brawn on a plate he did not touch it , for meat was for the father , when there was only enough for one . All of them had washed down their soup with a big bumper of fresh water , the good , clear drink of the fortnight 's end . " I have no beer , " said Maheude , when the father had seated himself in his turn . " I wanted to keep a little money . But if you would like some the little one can go and fetch a pint . " He looked at her in astonishment . What ! she had money , too ! " No , no , " he said , " I 've had a glass , it 's all right . " And Maheu began to swallow by slow spoonfuls the mixture of bread , potatoes , leeks , and sorrel piled up in the bowl which served him as a plate . Maheude , without putting Estelle down , helped Alzire to give him all that he requir
 ed , pushed near him the butter and the meat , and put his coffee on the fire to keep it quite hot . In the meanwhile , beside the fire , they began to wash themselves in the half of a barrel transformed into a tub . Catherine , whose turn came first , had filled it with warm water ; and she undressed herself tranquilly , took off her cap , her jacket , her breeches , and even her chemise , habituated to this since the age of eight , having grown up without seeing any harm in it . She only turned with her stomach to the fire , then rubbed herself vigorously with black soap . No one looked at her , even L�nore and Henri were no longer inquisitive to see how she was made . When she was clean she went up the stairs quite naked , leaving her damp chemise and other garments in a heap on the floor . But a quarrel broke out between the two brothers : Jeanlin had hastened to jump into the tub under the pretence that Zacharie was still eating ; and the latter hustled him , claiming his turn
  , and calling out that he was polite enough to allow Catherine to wash herself first , but he did not wish to have the rinsings of the young urchins , all the less since , when Jeanlin had been in , it would do to fill the school ink-pots . They ended by washing themselves together , also turning towards the fire , and they even helped each other , rubbing one another 's backs . Then , like their sister , they disappeared up the staircase naked . " What a slop they do make ! " murmured Maheude , taking up their garments from the floor to put them to dry . " Alzire , just sponge up a bit . " But a disturbance on the other side of the wall cut short her speech . One heard a man 's oaths , a woman 's crying , a whole stampede of battle , with hollow blows that sounded like thumps of an empty gourd . " Levaque 's wife is catching it , " Maheu peacefully stated as he scraped the bottom of his bowl with the spoon . " It 's queer ; Bouteloup made out that the soup was ready . " " Ah , yes
  ! ready , " said Maheude . " I saw the vegetables on the table , not even cleaned . " The cries redoubled , and there was a terrible push which shook the wall , followed by complete silence . Then the miner , swallowing the last spoonful , concluded , with an air of calm justice : " If the soup is not ready , one can understand . " And after having drunk a glassful of water , he attacked the brawn . He cut square pieces , stuck the point of his knife into them and ate them on his bread without a fork . There was no talking when the father was eating . He himself was hungry in silence ; he did not recognize the usual taste of Maigrat 's provisions ; this must come from somewhere else ; however , he put no question to his wife . He only asked if the old man was still sleeping upstairs . No , the grandfather had gone out for his usual walk . And there was silence again . But the odour of the meat made L�nore and Henri lift up their heads from the floor , where they were amusing thems
 elves with making rivulets with the spilt water . Both of them came and planted themselves near their father , the little one in front . Their eyes followed each morsel , full of hope when it set out from the plate and with an air of consternation when it was engulfed in the mouth . At last the father noticed the gluttonous desire which made their faces pale and their lips moist . " Have the children had any of it ? " he asked . And as his wife hesitated : " You know I do n't like injustice . It takes away my appetite when I see them there , begging for bits . " " But they 've had some of it , " she exclaimed , angrily . " If you were to listen to them you might give them your share and the others ' , too ; they would fill themselves till they burst . Is n't it true , Alzire , that we have all had some ? " " Sure enough , mother , " replied the little humpback , who under such circumstances could tell lies with the self-possession of a grown-up person . L�nore and Henri stood motio
 nless , shocked and rebellious at such lying , when they themselves were whipped if they did not tell the truth . Their little hearts began to swell , and they longed to protest , and to say that they , at all events , were not there when the others had some . " Get along with you , " said the mother , driving them to the other end of the room . " You ought to be ashamed of being always in your father 's plate ; and even if he was the only one to have any , does n't he work , while all you , a lot of good-for-nothings , ca n't do anything but spend ! Yes , and the more the bigger you are . " Maheu called them back . He seated L�nore on his left thigh , Henri on the right ; then he finished the brawn by playing at dinner with them . He cut small pieces , and each had his share . The children devoured with delight . When he had finished , he said to his wife : " No , do n't give me my coffee . I 'm going to wash first ; and just give me a hand to throw away this dirty water . " They 
 took hold of the handles of the tub and emptied it into the gutter before the door , when Jeanlin came down in dry garments , breeches and a woollen blouse , too large for him , which were weary of fading on his brother 's back . Seeing him slinking out through the open door , his mother stopped him . " Where are you off to ? " " Over there . " " Over where ? Listen to me . You go and gather a dandelion salad for this evening . Eh , do you hear ? If you do n't bring a salad back you 'll have to deal with me . " " All right ! " Jeanlin set out with hands in his pockets , trailing his sabots and slouching along , with his slender loins of a ten-year-old urchin , like an old miner . In his turn , Zacharie came down , more carefully dressed , his body covered by a black woollen knitted jacket with blue stripes . His father called out to him not to return late ; and he left , nodding his head with his pipe between his teeth , without replying . Again the tub was filled with warm water . 
 Maheu was already slowly taking off his jacket . At a look , Alzire led L�nore and Henri outside to play . The father did not like washing en famille , as was practised in many houses in the settlement . He blamed no one , however ; he simply said that it was good for the children to dabble together . " What are you doing up there ? " cried Maheude , up the staircase . " I 'm mending my dress that I tore yesterday , " replied Catherine . " All right . Do n't come down , your father is washing . " Then Maheu and Maheude were left alone . The latter decided to place Estelle on a chair , and by a miracle , finding herself near the fire the child did not scream , but turned towards her parents the vague eyes of a little creature without intelligence . He was crouching before the tub quite naked , having first plunged his head into it , well rubbed with that black soap the constant use of which discoloured and made yellow the hair of the race . Afterwards he got into the water , lathere
 d his chest , belly , arms , and thighs , scraping them energetically with both fists . His wife , standing by , watched him . " Well , then , " she began , " I saw your eyes when you came in . You were bothered , eh ? and it eased you , those provisions . Fancy ! those Piolaine people did n't give me a sou ! Oh ! they are kind enough ; they have dressed the little ones and I was ashamed to ask them , for it crosses me to ask for things . " She interrupted herself a moment to wedge Estelle into the chair lest she should tip over . The father continued to work away at his skin , without hastening by a question this story which interested him , patiently waiting for light . " I must tell you that Maigrat had refused me , oh ! straight ! like one kicks a dog out of doors . Guess if I was on a spree ! They keep you warm , woollen garments , but they do n't put anything into your stomach , eh ! " He lifted his head , still silent . Nothing at Piolaine , nothing at Maigrat 's : then where
  ? But , as usual , she was pulling up her sleeves to wash his back and those parts which he could not himself easily reach . Besides , he liked her to soap him , to rub him everywhere till she almost broke her wrists . She took soap and worked away at his shoulders while he held himself stiff so as to resist the shock . " Then I returned to Maigrat 's , and said to him , ah , I said something to him ! And that it did n't do to have no heart , and that evil would happen to him if there were any justice . That bothered him ; he turned his eyes and would like to have got away . " From the back she had got down to the buttocks and was pushing into the folds , not leaving any part of the body without passing over it , making him shine like her three saucepans on Saturdays after a big clean . Only she began to sweat with this tremendous exertion of her arms , so exhausted and out of breath that her words were choked . " At last he called me an old nuisance . We shall have bread until Sat
 urday , and the best is that he has lent me five francs . I have got butter , coffee , and chicory from him . I was even going to get the meat and potatoes there , only I saw that he was grumbling . Seven sous for the chitterlings , eighteen for the potatoes , and I 've got three francs seventy-five left for a ragout and a meat soup . Eh , I do n't think I 've wasted my morning ! " Now she began to wipe him , plugging with a towel the parts that would not dry . Feeling happy and without thinking of the future debt , he burst out laughing and took her in his arms . " Leave me alone , stupid ! You are damp , and wetting me . Only I 'm afraid Maigrat has ideas---- " She was about to speak of Catherine , but she stopped . What was the good of disturbing him ? It would only lead to endless discussion . " What ideas ? " he asked . " Why , ideas of robbing us . Catherine will have to examine the bill carefully . " He took her in his arms again , and this time did not let her go . The bath 
 always finished in this way : she enlivened him by the hard rubbing , and then by the towels which tickled the hairs of his arms and chest . Besides , among all his mates of the settlement it was the hour for stupidities , when more children were planted than were wanted . At night all the family were about . He pushed her towards the table , jesting like a worthy man who was enjoying the only good moment of the day , calling that taking his dessert , and a dessert which cost him nothing . She , with her loose figure and breast , struggled a little for fun . " You are stupid ! My Lord ! you are stupid ! And there 's Estelle looking at us . Wait till I turn her head . " " Oh , bosh ! at three months ; as if she understood ! " When he got up Maheu simply put on a dry pair of breeches . He liked , when he was clean and had taken his pleasure with his wife , to remain naked for a while . On his white skin , the whiteness of an anaemic girl , the scratches and gashes of the coal left tat
 too-marks , grafts as the miners called them ; and he was proud of them , and exhibited his big arms and broad chest shining like veined marble . In summer all the miners could be seen in this condition at their doors . He even went there for a moment now , in spite of the wet weather , and shouted out a rough joke to a comrade , whose breast was also naked , on the other side of the gardens . Others also appeared . And the children , trailing along the pathways , raised their heads and also laughed with delight at all this weary flesh of workers displayed in the open air . While drinking his coffee , without yet putting on a shirt , Maheu told his wife about the engineer 's anger over the planking . He was calm and unbent , and listened with a nod of approval to the sensible advice of Maheude , who showed much common sense in such affairs . She always repeated to him that nothing was gained by struggling against the Company . She afterwards told him about Madame Hennebeau 's visit 
 . Without saying so , both of them were proud of this . " Can I come down yet ? " asked Catherine , from the top of the staircase . " Yes , yes ; your father is drying himself . " The young girl had put on her Sunday dress , an old frock of rough blue poplin , already faded and worn in the folds . She had on a very simple bonnet of black tulle . " Hallo ! you 're dressed . Where are you going to ? " " I 'm going to Montsou to buy a ribbon for my bonnet . I 've taken off the old one ; it was too dirty . " " Have you got money , then ? " " No ! but Mouquette promised to lend me half a franc . " The mother let her go . But at the door she called her back . " Here ! do n't go and buy that ribbon at Maigrat 's . He will rob you , and he will think that we are rolling in wealth . " The father , who was crouching down before the fire to dry his neck and shoulders more quickly , contented himself with adding : " Try not to dawdle about at night on the road . " In the afternoon , Maheu worke
 d in his garden . Already he had sown there potatoes , beans , and peas ; and he now set about replanting cabbage and lettuce plants , which he had kept fresh from the night before . This bit of garden furnished them with vegetables , except potatoes of which they never had enough . He understood gardening very well , and could even grow artichokes , which was treated as sheer display by the neighbours . As he was preparing the bed , Levaque just then came out to smoke a pipe in his own square , looking at the cos lettuces which Bouteloup had planted in the morning ; for without the lodger 's energy in digging nothing would have grown there but nettles . And a conversation arose over the trellis . Levaque , refreshed and excited by thrashing his wife , vainly tried to take Maheu off to Rasseneur 's . Why , was he afraid of a glass ? They could have a game at skittles , lounge about for a while with the mates , and then come back to dinner . That was the way of life after leaving the
  pit . No doubt there was no harm in that , but Maheu was obstinate ; if he did not replant his lettuces they would be faded by to-morrow . In reality he refused out of good sense , not wishing to ask a farthing from his wife out of the change of the five-franc piece . Five o'clock was striking when Pierrone came to know if it was with Jeanlin that her Lydie had gone off . Levaque replied that it must be something of that sort , for B�bert had also disappeared , and those rascals always went prowling about together . When Maheu had quieted them by speaking of the dandelion salad , he and his comrade set about joking the young woman with the coarseness of good-natured devils . She was angry , but did not go away , in reality tickled by the strong words which made her scream with her hands to her sides . A lean woman came to her aid , stammering with anger like a clucking hen . Others in the distance on their doorsteps confided their alarms . Now the school was closed ; and all the c
 hildren were running about , there was a swarm of little creatures shouting and tumbling and fighting ; while those fathers who were not at the public-house were resting in groups of three or four , crouching on their heels as they did in the mine , smoking their pipes with an occasional word in the shelter of a wall . Pierronne went off in a fury when Levaque wanted to feel if her thighs were firm ; and he himself decided to go alone to Rasseneur 's , since Maheu was still planting . Twilight suddenly came on ; Maheude lit the lamp , irritated because neither her daughter nor the boys had come back . She could have guessed as much ; they never succeeded in taking together the only meal of the day at which it was possible for them to be all round the table . Then she was waiting for the dandelion salad . What could he be gathering at this hour , in this blackness of an oven , that nuisance of a child ! A salad would go so well with the stew which was simmering on the fire--potatoes 
 , leeks , sorrel , fricasseed with fried onion . The whole house smelt of that fried onion , that good odour which gets rank so soon , and which penetrates the bricks of the settlements with such infection that one perceives it far off in the country , the violent flavour of the poor man 's kitchen . Maheu , when he left the garden at nightfall , at once fell into a chair with his head against the wall . As soon as he sat down in the evening he went to sleep . The clock struck seven ; Henri and L�nore had just broken a plate in persisting in helping Alzire , who was laying the table , when Father Bonnemort came in first , in a hurry to dine and go back to the pit . Then Maheude woke up Maheu . " Come and eat ! So much the worse ! They are big enough to find the house . The nuisance is the salad ! " 
\ No newline at end of file


[36/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/48FictBronteC_JaneEyre_33_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/48FictBronteC_JaneEyre_33_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/48FictBronteC_JaneEyre_33_EN.txt.txt
new file mode 100644
index 0000000..36b6857
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/48FictBronteC_JaneEyre_33_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff CHAPTER XXXIII When Mr. St. John went , it was beginning to snow ; the whirling storm continued all night . The next day a keen wind brought fresh and blinding falls ; by twilight the valley was drifted up and almost impassable . I had closed my shutter , laid a mat to the door to prevent the snow from blowing in under it , trimmed my fire , and after sitting nearly an hour on the hearth listening to the muffled fury of the tempest , I lit a candle , took down " Marmion , " and beginning-- " Day set on Norham 's castled steep , And Tweed 's fair river broad and deep , And Cheviot 's mountains lone ; The massive towers , the donjon keep , The flanking walls that round them sweep , In yellow lustre shone"-- I soon forgot storm in music . I heard a noise : the wind , I thought , shook the door . No ; it was St. John Rivers , who , lifting the latch , came in out of the frozen hurricane--the howling darkness--and stood before me : the cloak that covered his tall figure all white as 
 a glacier . I was almost in consternation , so little had I expected any guest from the blocked-up vale that night . " Any ill news ? " I demanded . " Has anything happened ? " " No. How very easily alarmed you are ! " he answered , removing his cloak and hanging it up against the door , towards which he again coolly pushed the mat which his entrance had deranged . He stamped the snow from his boots . " I shall sully the purity of your floor , " said he , " but you must excuse me for once . " Then he approached the fire . " I have had hard work to get here , I assure you , " he observed , as he warmed his hands over the flame . " One drift took me up to the waist ; happily the snow is quite soft yet . " " But why are you come ? " I could not forbear saying . " Rather an inhospitable question to put to a visitor ; but since you ask it , I answer simply to have a little talk with you ; I got tired of my mute books and empty rooms . Besides , since yesterday I have experienced the exci
 tement of a person to whom a tale has been half-told , and who is impatient to hear the sequel . " He sat down . I recalled his singular conduct of yesterday , and really I began to fear his wits were touched . If he were insane , however , his was a very cool and collected insanity : I had never seen that handsome-featured face of his look more like chiselled marble than it did just now , as he put aside his snow-wet hair from his forehead and let the firelight shine free on his pale brow and cheek as pale , where it grieved me to discover the hollow trace of care or sorrow now so plainly graved . I waited , expecting he would say something I could at least comprehend ; but his hand was now at his chin , his finger on his lip : he was thinking . It struck me that his hand looked wasted like his face . A perhaps uncalled-for gush of pity came over my heart : I was moved to say-- " I wish Diana or Mary would come and live with you : it is too bad that you should be quite alone ; and 
 you are recklessly rash about your own health . " " Not at all , " said he : " I care for myself when necessary . I am well now . What do you see amiss in me ? " This was said with a careless , abstracted indifference , which showed that my solicitude was , at least in his opinion , wholly superfluous . I was silenced . He still slowly moved his finger over his upper lip , and still his eye dwelt dreamily on the glowing grate ; thinking it urgent to say something , I asked him presently if he felt any cold draught from the door , which was behind him . " No , no ! " he responded shortly and somewhat testily . " Well , " I reflected , " if you wo n't talk , you may be still ; I 'll let you alone now , and return to my book . " So I snuffed the candle and resumed the perusal of " Marmion . " He soon stirred ; my eye was instantly drawn to his movements ; he only took out a morocco pocket-book , thence produced a letter , which he read in silence , folded it , put it back , relapsed in
 to meditation . It was vain to try to read with such an inscrutable fixture before me ; nor could I , in impatience , consent to be dumb ; he might rebuff me if he liked , but talk I would . " Have you heard from Diana and Mary lately ? " " Not since the letter I showed you a week ago . " " There has not been any change made about your own arrangements ? You will not be summoned to leave England sooner than you expected ? " " I fear not , indeed : such chance is too good to befall me . " Baffled so far , I changed my ground . I bethought myself to talk about the school and my scholars . " Mary Garrett 's mother is better , and Mary came back to the school this morning , and I shall have four new girls next week from the Foundry Close--they would have come to-day but for the snow . " " Indeed ! " " Mr. Oliver pays for two . " " Does he ? " " He means to give the whole school a treat at Christmas . " " I know . " " Was it your suggestion ? " " No. " " Whose , then ? " " His daughter '
 s , I think . " " It is like her : she is so good-natured . " " Yes . " Again came the blank of a pause : the clock struck eight strokes . It aroused him ; he uncrossed his legs , sat erect , turned to me . " Leave your book a moment , and come a little nearer the fire , " he said . Wondering , and of my wonder finding no end , I complied . " Half-an-hour ago , " he pursued , " I spoke of my impatience to hear the sequel of a tale : on reflection , I find the matter will be better managed by my assuming the narrator 's part , and converting you into a listener . Before commencing , it is but fair to warn you that the story will sound somewhat hackneyed in your ears ; but stale details often regain a degree of freshness when they pass through new lips . For the rest , whether trite or novel , it is short . " Twenty years ago , a poor curate--never mind his name at this moment--fell in love with a rich man 's daughter ; she fell in love with him , and married him , against the advice 
 of all her friends , who consequently disowned her immediately after the wedding . Before two years passed , the rash pair were both dead , and laid quietly side by side under one slab . ( I have seen their grave ; it formed part of the pavement of a huge churchyard surrounding the grim , soot-black old cathedral of an overgrown manufacturing town in ---shire . ) They left a daughter , which , at its very birth , Charity received in her lap--cold as that of the snow-drift I almost stuck fast in to-night . Charity carried the friendless thing to the house of its rich maternal relations ; it was reared by an aunt-in- law , called ( I come to names now ) Mrs. Reed of Gateshead . You start--did you hear a noise ? I daresay it is only a rat scrambling along the rafters of the adjoining schoolroom : it was a barn before I had it repaired and altered , and barns are generally haunted by rats . --To proceed . Mrs. Reed kept the orphan ten years : whether it was happy or not with her , I can
 not say , never having been told ; but at the end of that time she transferred it to a place you know--being no other than Lowood School , where you so long resided yourself . It seems her career there was very honourable : from a pupil , she became a teacher , like yourself--really it strikes me there are parallel points in her history and yours--she left it to be a governess : there , again , your fates were analogous ; she undertook the education of the ward of a certain Mr. Rochester . " " Mr. Rivers ! " I interrupted . " I can guess your feelings , " he said , " but restrain them for a while : I have nearly finished ; hear me to the end . Of Mr. Rochester 's character I know nothing , but the one fact that he professed to offer honourable marriage to this young girl , and that at the very altar she discovered he had a wife yet alive , though a lunatic . What his subsequent conduct and proposals were is a matter of pure conjecture ; but when an event transpired which rendered in
 quiry after the governess necessary , it was discovered she was gone--no one could tell when , where , or how . She had left Thornfield Hall in the night ; every research after her course had been vain : the country had been scoured far and wide ; no vestige of information could be gathered respecting her . Yet that she should be found is become a matter of serious urgency : advertisements have been put in all the papers ; I myself have received a letter from one Mr. Briggs , a solicitor , communicating the details I have just imparted . Is it not an odd tale ? " " Just tell me this , " said I , " and since you know so much , you surely can tell it me--what of Mr. Rochester ? How and where is he ? What is he doing ? Is he well ? " " I am ignorant of all concerning Mr. Rochester : the letter never mentions him but to narrate the fraudulent and illegal attempt I have adverted to . You should rather ask the name of the governess--the nature of the event which requires her appearance . 
 " " Did no one go to Thornfield Hall , then ? Did no one see Mr. Rochester ? " " I suppose not . " " But they wrote to him ? " " Of course . " " And what did he say ? Who has his letters ? " " Mr. Briggs intimates that the answer to his application was not from Mr. Rochester , but from a lady : it is signed 'Alice Fairfax . ' " I felt cold and dismayed : my worst fears then were probably true : he had in all probability left England and rushed in reckless desperation to some former haunt on the Continent . And what opiate for his severe sufferings--what object for his strong passions--had he sought there ? I dared not answer the question . Oh , my poor master--once almost my husband--whom I had often called " my dear Edward ! " " He must have been a bad man , " observed Mr. Rivers . " You do n't know him--do n't pronounce an opinion upon him , " I said , with warmth . " Very well , " he answered quietly : " and indeed my head is otherwise occupied than with him : I have my tale to f
 inish . Since you wo n't ask the governess 's name , I must tell it of my own accord . Stay ! I have it here--it is always more satisfactory to see important points written down , fairly committed to black and white . " And the pocket-book was again deliberately produced , opened , sought through ; from one of its compartments was extracted a shabby slip of paper , hastily torn off : I recognised in its texture and its stains of ultra-marine , and lake , and vermillion , the ravished margin of the portrait-cover . He got up , held it close to my eyes : and I read , traced in Indian ink , in my own handwriting , the words " JANE EYRE"--the work doubtless of some moment of abstraction . " Briggs wrote to me of a Jane Eyre : " he said , " the advertisements demanded a Jane Eyre : I knew a Jane Elliott . --I confess I had my suspicions , but it was only yesterday afternoon they were at once resolved into certainty . You own the name and renounce the _alias_ ? " " Yes--yes ; but where is
  Mr. Briggs ? He perhaps knows more of Mr. Rochester than you do . " " Briggs is in London . I should doubt his knowing anything at all about Mr. Rochester ; it is not in Mr. Rochester he is interested . Meantime , you forget essential points in pursuing trifles : you do not inquire why Mr. Briggs sought after you--what he wanted with you . " " Well , what did he want ? " " Merely to tell you that your uncle , Mr. Eyre of Madeira , is dead ; that he has left you all his property , and that you are now rich--merely that--nothing more . " " I ! --rich ? " " Yes , you , rich--quite an heiress . " Silence succeeded . " You must prove your identity of course , " resumed St. John presently : " a step which will offer no difficulties ; you can then enter on immediate possession . Your fortune is vested in the English funds ; Briggs has the will and the necessary documents . " Here was a new card turned up ! It is a fine thing , reader , to be lifted in a moment from indigence to wealth--a 
 very fine thing ; but not a matter one can comprehend , or consequently enjoy , all at once . And then there are other chances in life far more thrilling and rapture-giving : _this_ is solid , an affair of the actual world , nothing ideal about it : all its associations are solid and sober , and its manifestations are the same . One does not jump , and spring , and shout hurrah ! at hearing one has got a fortune ; one begins to consider responsibilities , and to ponder business ; on a base of steady satisfaction rise certain grave cares , and we contain ourselves , and brood over our bliss with a solemn brow . Besides , the words Legacy , Bequest , go side by side with the words , Death , Funeral . My uncle I had heard was dead--my only relative ; ever since being made aware of his existence , I had cherished the hope of one day seeing him : now , I never should . And then this money came only to me : not to me and a rejoicing family , but to my isolated self . It was a grand boon d
 oubtless ; and independence would be glorious--yes , I felt that--that thought swelled my heart . " You unbend your forehead at last , " said Mr. Rivers . " I thought Medusa had looked at you , and that you were turning to stone . Perhaps now you will ask how much you are worth ? " " How much am I worth ? " " Oh , a trifle ! Nothing of course to speak of--twenty thousand pounds , I think they say--but what is that ? " " Twenty thousand pounds ? " Here was a new stunner--I had been calculating on four or five thousand . This news actually took my breath for a moment : Mr. St. John , whom I had never heard laugh before , laughed now . " Well , " said he , " if you had committed a murder , and I had told you your crime was discovered , you could scarcely look more aghast . " " It is a large sum--do n't you think there is a mistake ? " " No mistake at all . " " Perhaps you have read the figures wrong--it may be two thousand ! " " It is written in letters , not figures , --twenty thousan
 d . " I again felt rather like an individual of but average gastronomical powers sitting down to feast alone at a table spread with provisions for a hundred . Mr. Rivers rose now and put his cloak on . " If it were not such a very wild night , " he said , " I would send Hannah down to keep you company : you look too desperately miserable to be left alone . But Hannah , poor woman ! could not stride the drifts so well as I : her legs are not quite so long : so I must e'en leave you to your sorrows . Good-night . " He was lifting the latch : a sudden thought occurred to me . " Stop one minute ! " I cried . " Well ? " " It puzzles me to know why Mr. Briggs wrote to you about me ; or how he knew you , or could fancy that you , living in such an out-of-the-way place , had the power to aid in my discovery . " " Oh ! I am a clergyman , " he said ; " and the clergy are often appealed to about odd matters . " Again the latch rattled . " No ; that does not satisfy me ! " I exclaimed : and ind
 eed there was something in the hasty and unexplanatory reply which , instead of allaying , piqued my curiosity more than ever . " It is a very strange piece of business , " I added ; " I must know more about it . " " Another time . " " No ; to-night ! --to-night ! " and as he turned from the door , I placed myself between it and him . He looked rather embarrassed . " You certainly shall not go till you have told me all , " I said . " I would rather not just now . " " You shall ! --you must ! " " I would rather Diana or Mary informed you . " Of course these objections wrought my eagerness to a climax : gratified it must be , and that without delay ; and I told him so . " But I apprised you that I was a hard man , " said he , " difficult to persuade . " " And I am a hard woman , -- impossible to put off . " " And then , " he pursued , " I am cold : no fervour infects me . " " Whereas I am hot , and fire dissolves ice . The blaze there has thawed all the snow from your cloak ; by the s
 ame token , it has streamed on to my floor , and made it like a trampled street . As you hope ever to be forgiven , Mr. Rivers , the high crime and misdemeanour of spoiling a sanded kitchen , tell me what I wish to know . " " Well , then , " he said , " I yield ; if not to your earnestness , to your perseverance : as stone is worn by continual dropping . Besides , you must know some day , --as well now as later . Your name is Jane Eyre ? " " Of course : that was all settled before . " " You are not , perhaps , aware that I am your namesake ? --that I was christened St. John Eyre Rivers ? " " No , indeed ! I remember now seeing the letter E. comprised in your initials written in books you have at different times lent me ; but I never asked for what name it stood . But what then ? Surely-- " I stopped : I could not trust myself to entertain , much less to express , the thought that rushed upon me--that embodied itself , --that , in a second , stood out a strong , solid probability . C
 ircumstances knit themselves , fitted themselves , shot into order : the chain that had been lying hitherto a formless lump of links was drawn out straight , --every ring was perfect , the connection complete . I knew , by instinct , how the matter stood , before St. John had said another word ; but I cannot expect the reader to have the same intuitive perception , so I must repeat his explanation . " My mother 's name was Eyre ; she had two brothers ; one a clergyman , who married Miss Jane Reed , of Gateshead ; the other , John Eyre , Esq . , merchant , late of Funchal , Madeira . Mr. Briggs , being Mr. Eyre 's solicitor , wrote to us last August to inform us of our uncle 's death , and to say that he had left his property to his brother the clergyman 's orphan daughter , overlooking us , in consequence of a quarrel , never forgiven , between him and my father . He wrote again a few weeks since , to intimate that the heiress was lost , and asking if we knew anything of her . A nam
 e casually written on a slip of paper has enabled me to find her out . You know the rest . " Again he was going , but I set my back against the door . " Do let me speak , " I said ; " let me have one moment to draw breath and reflect . " I paused--he stood before me , hat in hand , looking composed enough . I resumed-- " Your mother was my father 's sister ? " " Yes . " " My aunt , consequently ? " He bowed . " My uncle John was your uncle John ? You , Diana , and Mary are his sister 's children , as I am his brother 's child ? " " Undeniably . " " You three , then , are my cousins ; half our blood on each side flows from the same source ? " " We are cousins ; yes . " I surveyed him . It seemed I had found a brother : one I could be proud of , --one I could love ; and two sisters , whose qualities were such , that , when I knew them but as mere strangers , they had inspired me with genuine affection and admiration . The two girls , on whom , kneeling down on the wet ground , and loo
 king through the low , latticed window of Moor House kitchen , I had gazed with so bitter a mixture of interest and despair , were my near kinswomen ; and the young and stately gentleman who had found me almost dying at his threshold was my blood relation . Glorious discovery to a lonely wretch ! This was wealth indeed ! --wealth to the heart ! --a mine of pure , genial affections . This was a blessing , bright , vivid , and exhilarating ; --not like the ponderous gift of gold : rich and welcome enough in its way , but sobering from its weight . I now clapped my hands in sudden joy--my pulse bounded , my veins thrilled . " Oh , I am glad ! --I am glad ! " I exclaimed . St. John smiled . " Did I not say you neglected essential points to pursue trifles ? " he asked . " You were serious when I told you you had got a fortune ; and now , for a matter of no moment , you are excited . " " What can you mean ? It may be of no moment to you ; you have sisters and do n't care for a cousin ; bu
 t I had nobody ; and now three relations , --or two , if you do n't choose to be counted , --are born into my world full-grown . I say again , I am glad ! " I walked fast through the room : I stopped , half suffocated with the thoughts that rose faster than I could receive , comprehend , settle them : --thoughts of what might , could , would , and should be , and that ere long . I looked at the blank wall : it seemed a sky thick with ascending stars , --every one lit me to a purpose or delight . Those who had saved my life , whom , till this hour , I had loved barrenly , I could now benefit . They were under a yoke , --I could free them : they were scattered , --I could reunite them : the independence , the affluence which was mine , might be theirs too . Were we not four ? Twenty thousand pounds shared equally would be five thousand each , justice--enough and to spare : justice would be done , --mutual happiness secured . Now the wealth did not weigh on me : now it was not a mere b
 equest of coin , --it was a legacy of life , hope , enjoyment . How I looked while these ideas were taking my spirit by storm , I cannot tell ; but I perceived soon that Mr. Rivers had placed a chair behind me , and was gently attempting to make me sit down on it . He also advised me to be composed ; I scorned the insinuation of helplessness and distraction , shook off his hand , and began to walk about again . " Write to Diana and Mary to-morrow , " I said , " and tell them to come home directly . Diana said they would both consider themselves rich with a thousand pounds , so with five thousand they will do very well . " " Tell me where I can get you a glass of water , " said St. John ; " you must really make an effort to tranquillise your feelings . " " Nonsense ! and what sort of an effect will the bequest have on you ? Will it keep you in England , induce you to marry Miss Oliver , and settle down like an ordinary mortal ? " " You wander : your head becomes confused . I have bee
 n too abrupt in communicating the news ; it has excited you beyond your strength . " " Mr. Rivers ! you quite put me out of patience : I am rational enough ; it is you who misunderstand , or rather who affect to misunderstand . " " Perhaps , if you explained yourself a little more fully , I should comprehend better . " " Explain ! What is there to explain ? You cannot fail to see that twenty thousand pounds , the sum in question , divided equally between the nephew and three nieces of our uncle , will give five thousand to each ? What I want is , that you should write to your sisters and tell them of the fortune that has accrued to them . " " To you , you mean . " " I have intimated my view of the case : I am incapable of taking any other . I am not brutally selfish , blindly unjust , or fiendishly ungrateful . Besides , I am resolved I will have a home and connections . I like Moor House , and I will live at Moor House ; I like Diana and Mary , and I will attach myself for life to 
 Diana and Mary . It would please and benefit me to have five thousand pounds ; it would torment and oppress me to have twenty thousand ; which , moreover , could never be mine in justice , though it might in law . I abandon to you , then , what is absolutely superfluous to me . Let there be no opposition , and no discussion about it ; let us agree amongst each other , and decide the point at once . " " This is acting on first impulses ; you must take days to consider such a matter , ere your word can be regarded as valid . " " Oh ! if all you doubt is my sincerity , I am easy : you see the justice of the case ? " " I _do_ see a certain justice ; but it is contrary to all custom . Besides , the entire fortune is your right : my uncle gained it by his own efforts ; he was free to leave it to whom he would : he left it to you . After all , justice permits you to keep it : you may , with a clear conscience , consider it absolutely your own . " " With me , " said I , " it is fully as muc
 h a matter of feeling as of conscience : I must indulge my feelings ; I so seldom have had an opportunity of doing so . Were you to argue , object , and annoy me for a year , I could not forego the delicious pleasure of which I have caught a glimpse--that of repaying , in part , a mighty obligation , and winning to myself lifelong friends . " " You think so now , " rejoined St. John , " because you do not know what it is to possess , nor consequently to enjoy wealth : you cannot form a notion of the importance twenty thousand pounds would give you ; of the place it would enable you to take in society ; of the prospects it would open to you : you cannot-- " " And you , " I interrupted , " cannot at all imagine the craving I have for fraternal and sisterly love . I never had a home , I never had brothers or sisters ; I must and will have them now : you are not reluctant to admit me and own me , are you ? " " Jane , I will be your brother--my sisters will be your sisters--without stipu
 lating for this sacrifice of your just rights . " " Brother ? Yes ; at the distance of a thousand leagues ! Sisters ? Yes ; slaving amongst strangers ! I , wealthy--gorged with gold I never earned and do not merit ! You , penniless ! Famous equality and fraternisation ! Close union ! Intimate attachment ! " " But , Jane , your aspirations after family ties and domestic happiness may be realised otherwise than by the means you contemplate : you may marry . " " Nonsense , again ! Marry ! I do n't want to marry , and never shall marry . " " That is saying too much : such hazardous affirmations are a proof of the excitement under which you labour . " " It is not saying too much : I know what I feel , and how averse are my inclinations to the bare thought of marriage . No one would take me for love ; and I will not be regarded in the light of a mere money speculation . And I do not want a stranger--unsympathising , alien , different from me ; I want my kindred : those with whom I have fu
 ll fellow- feeling . Say again you will be my brother : when you uttered the words I was satisfied , happy ; repeat them , if you can , repeat them sincerely . " " I think I can . I know I have always loved my own sisters ; and I know on what my affection for them is grounded , --respect for their worth and admiration of their talents . You too have principle and mind : your tastes and habits resemble Diana 's and Mary 's ; your presence is always agreeable to me ; in your conversation I have already for some time found a salutary solace . I feel I can easily and naturally make room in my heart for you , as my third and youngest sister . " " Thank you : that contents me for to-night . Now you had better go ; for if you stay longer , you will perhaps irritate me afresh by some mistrustful scruple . " " And the school , Miss Eyre ? It must now be shut up , I suppose ? " " No. I will retain my post of mistress till you get a substitute . " He smiled approbation : we shook hands , and h
 e took leave . I need not narrate in detail the further struggles I had , and arguments I used , to get matters regarding the legacy settled as I wished . My task was a very hard one ; but , as I was absolutely resolved--as my cousins saw at length that my mind was really and immutably fixed on making a just division of the property--as they must in their own hearts have felt the equity of the intention ; and must , besides , have been innately conscious that in my place they would have done precisely what I wished to do--they yielded at length so far as to consent to put the affair to arbitration . The judges chosen were Mr. Oliver and an able lawyer : both coincided in my opinion : I carried my point . The instruments of transfer were drawn out : St. John , Diana , Mary , and I , each became possessed of a competency . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/49FictBronteC_JaneEyre_6_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/49FictBronteC_JaneEyre_6_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/49FictBronteC_JaneEyre_6_EN.txt.txt
new file mode 100644
index 0000000..c5ba6dc
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/49FictBronteC_JaneEyre_6_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff CHAPTER VI The next day commenced as before , getting up and dressing by rushlight ; but this morning we were obliged to dispense with the ceremony of washing ; the water in the pitchers was frozen . A change had taken place in the weather the preceding evening , and a keen north-east wind , whistling through the crevices of our bedroom windows all night long , had made us shiver in our beds , and turned the contents of the ewers to ice . Before the long hour and a half of prayers and Bible-reading was over , I felt ready to perish with cold . Breakfast-time came at last , and this morning the porridge was not burnt ; the quality was eatable , the quantity small . How small my portion seemed ! I wished it had been doubled . In the course of the day I was enrolled a member of the fourth class , and regular tasks and occupations were assigned me : hitherto , I had only been a spectator of the proceedings at Lowood ; I was now to become an actor therein . At first , being little ac
 customed to learn by heart , the lessons appeared to me both long and difficult ; the frequent change from task to task , too , bewildered me ; and I was glad when , about three o'clock in the afternoon , Miss Smith put into my hands a border of muslin two yards long , together with needle , thimble , &c . , and sent me to sit in a quiet corner of the schoolroom , with directions to hem the same . At that hour most of the others were sewing likewise ; but one class still stood round Miss Scatcherd 's chair reading , and as all was quiet , the subject of their lessons could be heard , together with the manner in which each girl acquitted herself , and the animadversions or commendations of Miss Scatcherd on the performance . It was English history : among the readers I observed my acquaintance of the verandah : at the commencement of the lesson , her place had been at the top of the class , but for some error of pronunciation , or some inattention to stops , she was suddenly sent to 
 the very bottom . Even in that obscure position , Miss Scatcherd continued to make her an object of constant notice : she was continually addressing to her such phrases as the following : -- " Burns " ( such it seems was her name : the girls here were all called by their surnames , as boys are elsewhere ) , " Burns , you are standing on the side of your shoe ; turn your toes out immediately . " " Burns , you poke your chin most unpleasantly ; draw it in . " " Burns , I insist on your holding your head up ; I will not have you before me in that attitude , " &c . &c . A chapter having been read through twice , the books were closed and the girls examined . The lesson had comprised part of the reign of Charles I. , and there were sundry questions about tonnage and poundage and ship- money , which most of them appeared unable to answer ; still , every little difficulty was solved instantly when it reached Burns : her memory seemed to have retained the substance of the whole lesson , and
  she was ready with answers on every point . I kept expecting that Miss Scatcherd would praise her attention ; but , instead of that , she suddenly cried out-- " You dirty , disagreeable girl ! you have never cleaned your nails this morning ! " Burns made no answer : I wondered at her silence . " Why , " thought I , " does she not explain that she could neither clean her nails nor wash her face , as the water was frozen ? " My attention was now called off by Miss Smith desiring me to hold a skein of thread : while she was winding it , she talked to me from time to time , asking whether I had ever been at school before , whether I could mark , stitch , knit , &c . ; till she dismissed me , I could not pursue my observations on Miss Scatcherd 's movements . When I returned to my seat , that lady was just delivering an order of which I did not catch the import ; but Burns immediately left the class , and going into the small inner room where the books were kept , returned in half a min
 ute , carrying in her hand a bundle of twigs tied together at one end . This ominous tool she presented to Miss Scatcherd with a respectful curtesy ; then she quietly , and without being told , unloosed her pinafore , and the teacher instantly and sharply inflicted on her neck a dozen strokes with the bunch of twigs . Not a tear rose to Burns ' eye ; and , while I paused from my sewing , because my fingers quivered at this spectacle with a sentiment of unavailing and impotent anger , not a feature of her pensive face altered its ordinary expression . " Hardened girl ! " exclaimed Miss Scatcherd ; " nothing can correct you of your slatternly habits : carry the rod away . " Burns obeyed : I looked at her narrowly as she emerged from the book-closet ; she was just putting back her handkerchief into her pocket , and the trace of a tear glistened on her thin cheek . The play-hour in the evening I thought the pleasantest fraction of the day at Lowood : the bit of bread , the draught of co
 ffee swallowed at five o'clock had revived vitality , if it had not satisfied hunger : the long restraint of the day was slackened ; the schoolroom felt warmer than in the morning--its fires being allowed to burn a little more brightly , to supply , in some measure , the place of candles , not yet introduced : the ruddy gloaming , the licensed uproar , the confusion of many voices gave one a welcome sense of liberty . On the evening of the day on which I had seen Miss Scatcherd flog her pupil , Burns , I wandered as usual among the forms and tables and laughing groups without a companion , yet not feeling lonely : when I passed the windows , I now and then lifted a blind , and looked out ; it snowed fast , a drift was already forming against the lower panes ; putting my ear close to the window , I could distinguish from the gleeful tumult within , the disconsolate moan of the wind outside . Probably , if I had lately left a good home and kind parents , this would have been the hour 
 when I should most keenly have regretted the separation ; that wind would then have saddened my heart ; this obscure chaos would have disturbed my peace ! as it was , I derived from both a strange excitement , and reckless and feverish , I wished the wind to howl more wildly , the gloom to deepen to darkness , and the confusion to rise to clamour . Jumping over forms , and creeping under tables , I made my way to one of the fire-places ; there , kneeling by the high wire fender , I found Burns , absorbed , silent , abstracted from all round her by the companionship of a book , which she read by the dim glare of the embers . " Is it still 'Rasselas ' ? " I asked , coming behind her . " Yes , " she said , " and I have just finished it . " And in five minutes more she shut it up . I was glad of this . " Now , " thought I , " I can perhaps get her to talk . " I sat down by her on the floor . " What is your name besides Burns ? " " Helen . " " Do you come a long way from here ? " " I com
 e from a place farther north , quite on the borders of Scotland . " " Will you ever go back ? " " I hope so ; but nobody can be sure of the future . " " You must wish to leave Lowood ? " " No ! why should I ? I was sent to Lowood to get an education ; and it would be of no use going away until I have attained that object . " " But that teacher , Miss Scatcherd , is so cruel to you ? " " Cruel ? Not at all ! She is severe : she dislikes my faults . " " And if I were in your place I should dislike her ; I should resist her . If she struck me with that rod , I should get it from her hand ; I should break it under her nose . " " Probably you would do nothing of the sort : but if you did , Mr. Brocklehurst would expel you from the school ; that would be a great grief to your relations . It is far better to endure patiently a smart which nobody feels but yourself , than to commit a hasty action whose evil consequences will extend to all connected with you ; and besides , the Bible bids us
  return good for evil . " " But then it seems disgraceful to be flogged , and to be sent to stand in the middle of a room full of people ; and you are such a great girl : I am far younger than you , and I could not bear it . " " Yet it would be your duty to bear it , if you could not avoid it : it is weak and silly to say you _cannot bear_ what it is your fate to be required to bear . " I heard her with wonder : I could not comprehend this doctrine of endurance ; and still less could I understand or sympathise with the forbearance she expressed for her chastiser . Still I felt that Helen Burns considered things by a light invisible to my eyes . I suspected she might be right and I wrong ; but I would not ponder the matter deeply ; like Felix , I put it off to a more convenient season . " You say you have faults , Helen : what are they ? To me you seem very good . " " Then learn from me , not to judge by appearances : I am , as Miss Scatcherd said , slatternly ; I seldom put , and ne
 ver keep , things , in order ; I am careless ; I forget rules ; I read when I should learn my lessons ; I have no method ; and sometimes I say , like you , I cannot _bear_ to be subjected to systematic arrangements . This is all very provoking to Miss Scatcherd , who is naturally neat , punctual , and particular . " " And cross and cruel , " I added ; but Helen Burns would not admit my addition : she kept silence . " Is Miss Temple as severe to you as Miss Scatcherd ? " At the utterance of Miss Temple 's name , a soft smile flitted over her grave face . " Miss Temple is full of goodness ; it pains her to be severe to any one , even the worst in the school : she sees my errors , and tells me of them gently ; and , if I do anything worthy of praise , she gives me my meed liberally . One strong proof of my wretchedly defective nature is , that even her expostulations , so mild , so rational , have not influence to cure me of my faults ; and even her praise , though I value it most high
 ly , cannot stimulate me to continued care and foresight . " " That is curious , " said I , " it is so easy to be careful . " " For _you_ I have no doubt it is . I observed you in your class this morning , and saw you were closely attentive : your thoughts never seemed to wander while Miss Miller explained the lesson and questioned you . Now , mine continually rove away ; when I should be listening to Miss Scatcherd , and collecting all she says with assiduity , often I lose the very sound of her voice ; I fall into a sort of dream . Sometimes I think I am in Northumberland , and that the noises I hear round me are the bubbling of a little brook which runs through Deepden , near our house ; --then , when it comes to my turn to reply , I have to be awakened ; and having heard nothing of what was read for listening to the visionary brook , I have no answer ready . " " Yet how well you replied this afternoon . " " It was mere chance ; the subject on which we had been reading had intere
 sted me . This afternoon , instead of dreaming of Deepden , I was wondering how a man who wished to do right could act so unjustly and unwisely as Charles the First sometimes did ; and I thought what a pity it was that , with his integrity and conscientiousness , he could see no farther than the prerogatives of the crown . If he had but been able to look to a distance , and see how what they call the spirit of the age was tending ! Still , I like Charles--I respect him--I pity him , poor murdered king ! Yes , his enemies were the worst : they shed blood they had no right to shed . How dared they kill him ! " Helen was talking to herself now : she had forgotten I could not very well understand her--that I was ignorant , or nearly so , of the subject she discussed . I recalled her to my level . " And when Miss Temple teaches you , do your thoughts wander then ? " " No , certainly , not often ; because Miss Temple has generally something to say which is newer than my own reflections ; 
 her language is singularly agreeable to me , and the information she communicates is often just what I wished to gain . " " Well , then , with Miss Temple you are good ? " " Yes , in a passive way : I make no effort ; I follow as inclination guides me . There is no merit in such goodness . " " A great deal : you are good to those who are good to you . It is all I ever desire to be . If people were always kind and obedient to those who are cruel and unjust , the wicked people would have it all their own way : they would never feel afraid , and so they would never alter , but would grow worse and worse . When we are struck at without a reason , we should strike back again very hard ; I am sure we should--so hard as to teach the person who struck us never to do it again . " " You will change your mind , I hope , when you grow older : as yet you are but a little untaught girl . " " But I feel this , Helen ; I must dislike those who , whatever I do to please them , persist in disliking m
 e ; I must resist those who punish me unjustly . It is as natural as that I should love those who show me affection , or submit to punishment when I feel it is deserved . " " Heathens and savage tribes hold that doctrine , but Christians and civilised nations disown it . " " How ? I do n't understand . " " It is not violence that best overcomes hate--nor vengeance that most certainly heals injury . " " What then ? " " Read the New Testament , and observe what Christ says , and how He acts ; make His word your rule , and His conduct your example . " " What does He say ? " " Love your enemies ; bless them that curse you ; do good to them that hate you and despitefully use you . " " Then I should love Mrs. Reed , which I cannot do ; I should bless her son John , which is impossible . " In her turn , Helen Burns asked me to explain , and I proceeded forthwith to pour out , in my own way , the tale of my sufferings and resentments . Bitter and truculent when excited , I spoke as I felt ,
  without reserve or softening . Helen heard me patiently to the end : I expected she would then make a remark , but she said nothing . " Well , " I asked impatiently , " is not Mrs. Reed a hard-hearted , bad woman ? " " She has been unkind to you , no doubt ; because you see , she dislikes your cast of character , as Miss Scatcherd does mine ; but how minutely you remember all she has done and said to you ! What a singularly deep impression her injustice seems to have made on your heart ! No ill-usage so brands its record on my feelings . Would you not be happier if you tried to forget her severity , together with the passionate emotions it excited ? Life appears to me too short to be spent in nursing animosity or registering wrongs . We are , and must be , one and all , burdened with faults in this world : but the time will soon come when , I trust , we shall put them off in putting off our corruptible bodies ; when debasement and sin will fall from us with this cumbrous frame of f
 lesh , and only the spark of the spirit will remain , --the impalpable principle of light and thought , pure as when it left the Creator to inspire the creature : whence it came it will return ; perhaps again to be communicated to some being higher than man--perhaps to pass through gradations of glory , from the pale human soul to brighten to the seraph ! Surely it will never , on the contrary , be suffered to degenerate from man to fiend ? No ; I cannot believe that : I hold another creed : which no one ever taught me , and which I seldom mention ; but in which I delight , and to which I cling : for it extends hope to all : it makes Eternity a rest--a mighty home , not a terror and an abyss . Besides , with this creed , I can so clearly distinguish between the criminal and his crime ; I can so sincerely forgive the first while I abhor the last : with this creed revenge never worries my heart , degradation never too deeply disgusts me , injustice never crushes me too low : I live in
  calm , looking to the end . " Helen 's head , always drooping , sank a little lower as she finished this sentence . I saw by her look she wished no longer to talk to me , but rather to converse with her own thoughts . She was not allowed much time for meditation : a monitor , a great rough girl , presently came up , exclaiming in a strong Cumberland accent-- " Helen Burns , if you do n't go and put your drawer in order , and fold up your work this minute , I 'll tell Miss Scatcherd to come and look at it ! " Helen sighed as her reverie fled , and getting up , obeyed the monitor without reply as without delay . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/50FictCarrollL_Wonderland_2_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/50FictCarrollL_Wonderland_2_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/50FictCarrollL_Wonderland_2_EN.txt.txt
new file mode 100644
index 0000000..07bb254
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/50FictCarrollL_Wonderland_2_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff CHAPTER II . The Pool of Tears 'Curiouser and curiouser ! ' cried Alice ( she was so much surprised , that for the moment she quite forgot how to speak good English ) ; 'now I 'm opening out like the largest telescope that ever was ! Good-bye , feet ! ' ( for when she looked down at her feet , they seemed to be almost out of sight , they were getting so far off ) . 'Oh , my poor little feet , I wonder who will put on your shoes and stockings for you now , dears ? I 'm sure _I_ sha n't be able ! I shall be a great deal too far off to trouble myself about you : you must manage the best way you can ; --but I must be kind to them , ' thought Alice , 'or perhaps they wo n't walk the way I want to go ! Let me see : I 'll give them a new pair of boots every Christmas . ' And she went on planning to herself how she would manage it . 'They must go by the carrier , ' she thought ; 'and how funny it 'll seem , sending presents to one 's own feet ! And how odd the directions will look ! ALI
 CE'S RIGHT FOOT , ESQ . HEARTHRUG , NEAR THE FENDER , ( WITH ALICE'S LOVE ) . Oh dear , what nonsense I 'm talking ! ' Just then her head struck against the roof of the hall : in fact she was now more than nine feet high , and she at once took up the little golden key and hurried off to the garden door . Poor Alice ! It was as much as she could do , lying down on one side , to look through into the garden with one eye ; but to get through was more hopeless than ever : she sat down and began to cry again . 'You ought to be ashamed of yourself , ' said Alice , 'a great girl like you , ' ( she might well say this ) , 'to go on crying in this way ! Stop this moment , I tell you ! ' But she went on all the same , shedding gallons of tears , until there was a large pool all round her , about four inches deep and reaching half down the hall . After a time she heard a little pattering of feet in the distance , and she hastily dried her eyes to see what was coming . It was the White Rabbit r
 eturning , splendidly dressed , with a pair of white kid gloves in one hand and a large fan in the other : he came trotting along in a great hurry , muttering to himself as he came , 'Oh ! the Duchess , the Duchess ! Oh ! wo n't she be savage if I 've kept her waiting ! ' Alice felt so desperate that she was ready to ask help of any one ; so , when the Rabbit came near her , she began , in a low , timid voice , 'If you please , sir-- ' The Rabbit started violently , dropped the white kid gloves and the fan , and skurried away into the darkness as hard as he could go . Alice took up the fan and gloves , and , as the hall was very hot , she kept fanning herself all the time she went on talking : 'Dear , dear ! How queer everything is to-day ! And yesterday things went on just as usual . I wonder if I 've been changed in the night ? Let me think : was I the same when I got up this morning ? I almost think I can remember feeling a little different . But if I 'm not the same , the next q
 uestion is , Who in the world am I ? Ah , THAT'S the great puzzle ! ' And she began thinking over all the children she knew that were of the same age as herself , to see if she could have been changed for any of them . 'I 'm sure I 'm not Ada , ' she said , 'for her hair goes in such long ringlets , and mine does n't go in ringlets at all ; and I 'm sure I ca n't be Mabel , for I know all sorts of things , and she , oh ! she knows such a very little ! Besides , SHE'S she , and I 'm I , and--oh dear , how puzzling it all is ! I 'll try if I know all the things I used to know . Let me see : four times five is twelve , and four times six is thirteen , and four times seven is--oh dear ! I shall never get to twenty at that rate ! However , the Multiplication Table does n't signify : let 's try Geography . London is the capital of Paris , and Paris is the capital of Rome , and Rome--no , THAT'S all wrong , I 'm certain ! I must have been changed for Mabel ! I 'll try and say " How doth th
 e little-- " ' and she crossed her hands on her lap as if she were saying lessons , and began to repeat it , but her voice sounded hoarse and strange , and the words did not come the same as they used to do : -- 'How doth the little crocodile Improve his shining tail , And pour the waters of the Nile On every golden scale ! 'How cheerfully he seems to grin , How neatly spread his claws , And welcome little fishes in With gently smiling jaws ! ' 'I 'm sure those are not the right words , ' said poor Alice , and her eyes filled with tears again as she went on , 'I must be Mabel after all , and I shall have to go and live in that poky little house , and have next to no toys to play with , and oh ! ever so many lessons to learn ! No , I 've made up my mind about it ; if I 'm Mabel , I 'll stay down here ! It 'll be no use their putting their heads down and saying " Come up again , dear ! " I shall only look up and say " Who am I then ? Tell me that first , and then , if I like being tha
 t person , I 'll come up : if not , I 'll stay down here till I 'm somebody else"--but , oh dear ! ' cried Alice , with a sudden burst of tears , 'I do wish they WOULD put their heads down ! I am so VERY tired of being all alone here ! ' As she said this she looked down at her hands , and was surprised to see that she had put on one of the Rabbit 's little white kid gloves while she was talking . 'How CAN I have done that ? ' she thought . 'I must be growing small again . ' She got up and went to the table to measure herself by it , and found that , as nearly as she could guess , she was now about two feet high , and was going on shrinking rapidly : she soon found out that the cause of this was the fan she was holding , and she dropped it hastily , just in time to avoid shrinking away altogether . 'That WAS a narrow escape ! ' said Alice , a good deal frightened at the sudden change , but very glad to find herself still in existence ; 'and now for the garden ! ' and she ran with all
  speed back to the little door : but , alas ! the little door was shut again , and the little golden key was lying on the glass table as before , 'and things are worse than ever , ' thought the poor child , 'for I never was so small as this before , never ! And I declare it 's too bad , that it is ! ' As she said these words her foot slipped , and in another moment , splash ! she was up to her chin in salt water . Her first idea was that she had somehow fallen into the sea , 'and in that case I can go back by railway , ' she said to herself . ( Alice had been to the seaside once in her life , and had come to the general conclusion , that wherever you go to on the English coast you find a number of bathing machines in the sea , some children digging in the sand with wooden spades , then a row of lodging houses , and behind them a railway station . ) However , she soon made out that she was in the pool of tears which she had wept when she was nine feet high . 'I wish I had n't cried s
 o much ! ' said Alice , as she swam about , trying to find her way out . 'I shall be punished for it now , I suppose , by being drowned in my own tears ! That WILL be a queer thing , to be sure ! However , everything is queer to-day . ' Just then she heard something splashing about in the pool a little way off , and she swam nearer to make out what it was : at first she thought it must be a walrus or hippopotamus , but then she remembered how small she was now , and she soon made out that it was only a mouse that had slipped in like herself . 'Would it be of any use , now , ' thought Alice , 'to speak to this mouse ? Everything is so out-of-the-way down here , that I should think very likely it can talk : at any rate , there 's no harm in trying . ' So she began : 'O Mouse , do you know the way out of this pool ? I am very tired of swimming about here , O Mouse ! ' ( Alice thought this must be the right way of speaking to a mouse : she had never done such a thing before , but she re
 membered having seen in her brother 's Latin Grammar , 'A mouse--of a mouse--to a mouse--a mouse--O mouse ! ' ) The Mouse looked at her rather inquisitively , and seemed to her to wink with one of its little eyes , but it said nothing . 'Perhaps it does n't understand English , ' thought Alice ; 'I daresay it 's a French mouse , come over with William the Conqueror . ' ( For , with all her knowledge of history , Alice had no very clear notion how long ago anything had happened . ) So she began again : 'Ou est ma chatte ? ' which was the first sentence in her French lesson-book . The Mouse gave a sudden leap out of the water , and seemed to quiver all over with fright . 'Oh , I beg your pardon ! ' cried Alice hastily , afraid that she had hurt the poor animal 's feelings . 'I quite forgot you did n't like cats . ' 'Not like cats ! ' cried the Mouse , in a shrill , passionate voice . 'Would YOU like cats if you were me ? ' 'Well , perhaps not , ' said Alice in a soothing tone : 'do n'
 t be angry about it . And yet I wish I could show you our cat Dinah : I think you 'd take a fancy to cats if you could only see her . She is such a dear quiet thing , ' Alice went on , half to herself , as she swam lazily about in the pool , 'and she sits purring so nicely by the fire , licking her paws and washing her face--and she is such a nice soft thing to nurse--and she 's such a capital one for catching mice--oh , I beg your pardon ! ' cried Alice again , for this time the Mouse was bristling all over , and she felt certain it must be really offended . 'We wo n't talk about her any more if you 'd rather not . ' 'We indeed ! ' cried the Mouse , who was trembling down to the end of his tail . 'As if I would talk on such a subject ! Our family always HATED cats : nasty , low , vulgar things ! Do n't let me hear the name again ! ' 'I wo n't indeed ! ' said Alice , in a great hurry to change the subject of conversation . 'Are you--are you fond--of--of dogs ? ' The Mouse did not an
 swer , so Alice went on eagerly : 'There is such a nice little dog near our house I should like to show you ! A little bright-eyed terrier , you know , with oh , such long curly brown hair ! And it 'll fetch things when you throw them , and it 'll sit up and beg for its dinner , and all sorts of things--I ca n't remember half of them--and it belongs to a farmer , you know , and he says it 's so useful , it 's worth a hundred pounds ! He says it kills all the rats and--oh dear ! ' cried Alice in a sorrowful tone , 'I 'm afraid I 've offended it again ! ' For the Mouse was swimming away from her as hard as it could go , and making quite a commotion in the pool as it went . So she called softly after it , 'Mouse dear ! Do come back again , and we wo n't talk about cats or dogs either , if you do n't like them ! ' When the Mouse heard this , it turned round and swam slowly back to her : its face was quite pale ( with passion , Alice thought ) , and it said in a low trembling voice , 'Le
 t us get to the shore , and then I 'll tell you my history , and you 'll understand why it is I hate cats and dogs . ' It was high time to go , for the pool was getting quite crowded with the birds and animals that had fallen into it : there were a Duck and a Dodo , a Lory and an Eaglet , and several other curious creatures . Alice led the way , and the whole party swam to the shore . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/51FictCarrollL_Wonderland_5_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/51FictCarrollL_Wonderland_5_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/51FictCarrollL_Wonderland_5_EN.txt.txt
new file mode 100644
index 0000000..172a925
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/51FictCarrollL_Wonderland_5_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff CHAPTER V. Advice from a Caterpillar The Caterpillar and Alice looked at each other for some time in silence : at last the Caterpillar took the hookah out of its mouth , and addressed her in a languid , sleepy voice . 'Who are YOU ? ' said the Caterpillar . This was not an encouraging opening for a conversation . Alice replied , rather shyly , 'I--I hardly know , sir , just at present--at least I know who I WAS when I got up this morning , but I think I must have been changed several times since then . ' 'What do you mean by that ? ' said the Caterpillar sternly . 'Explain yourself ! ' 'I ca n't explain MYSELF , I 'm afraid , sir ' said Alice , 'because I 'm not myself , you see . ' 'I do n't see , ' said the Caterpillar . 'I 'm afraid I ca n't put it more clearly , ' Alice replied very politely , 'for I ca n't understand it myself to begin with ; and being so many different sizes in a day is very confusing . ' 'It is n't , ' said the Caterpillar . 'Well , perhaps you have n't f
 ound it so yet , ' said Alice ; 'but when you have to turn into a chrysalis--you will some day , you know--and then after that into a butterfly , I should think you 'll feel it a little queer , wo n't you ? ' 'Not a bit , ' said the Caterpillar . 'Well , perhaps your feelings may be different , ' said Alice ; 'all I know is , it would feel very queer to ME . ' 'You ! ' said the Caterpillar contemptuously . 'Who are YOU ? ' Which brought them back again to the beginning of the conversation . Alice felt a little irritated at the Caterpillar 's making such VERY short remarks , and she drew herself up and said , very gravely , 'I think , you ought to tell me who YOU are , first . ' 'Why ? ' said the Caterpillar . Here was another puzzling question ; and as Alice could not think of any good reason , and as the Caterpillar seemed to be in a VERY unpleasant state of mind , she turned away . 'Come back ! ' the Caterpillar called after her . 'I 've something important to say ! ' This sounded
  promising , certainly : Alice turned and came back again . 'Keep your temper , ' said the Caterpillar . 'Is that all ? ' said Alice , swallowing down her anger as well as she could . 'No , ' said the Caterpillar . Alice thought she might as well wait , as she had nothing else to do , and perhaps after all it might tell her something worth hearing . For some minutes it puffed away without speaking , but at last it unfolded its arms , took the hookah out of its mouth again , and said , 'So you think you 're changed , do you ? ' 'I 'm afraid I am , sir , ' said Alice ; 'I ca n't remember things as I used--and I do n't keep the same size for ten minutes together ! ' 'Ca n't remember WHAT things ? ' said the Caterpillar . 'Well , I 've tried to say " HOW DOTH THE LITTLE BUSY BEE , " but it all came different ! ' Alice replied in a very melancholy voice . 'Repeat , " YOU ARE OLD , FATHER WILLIAM , " ' said the Caterpillar . Alice folded her hands , and began : -- 'You are old , Father Wi
 lliam , ' the young man said , 'And your hair has become very white ; And yet you incessantly stand on your head-- Do you think , at your age , it is right ? ' 'In my youth , ' Father William replied to his son , 'I feared it might injure the brain ; But , now that I 'm perfectly sure I have none , Why , I do it again and again . ' 'You are old , ' said the youth , 'as I mentioned before , And have grown most uncommonly fat ; Yet you turned a back-somersault in at the door-- Pray , what is the reason of that ? ' 'In my youth , ' said the sage , as he shook his grey locks , 'I kept all my limbs very supple By the use of this ointment--one shilling the box-- Allow me to sell you a couple ? ' 'You are old , ' said the youth , 'and your jaws are too weak For anything tougher than suet ; Yet you finished the goose , with the bones and the beak-- Pray how did you manage to do it ? ' 'In my youth , ' said his father , 'I took to the law , And argued each case with my wife ; And the muscula
 r strength , which it gave to my jaw , Has lasted the rest of my life . ' 'You are old , ' said the youth , 'one would hardly suppose That your eye was as steady as ever ; Yet you balanced an eel on the end of your nose-- What made you so awfully clever ? ' 'I have answered three questions , and that is enough , ' Said his father ; 'do n't give yourself airs ! Do you think I can listen all day to such stuff ? Be off , or I 'll kick you down stairs ! ' 'That is not said right , ' said the Caterpillar . 'Not QUITE right , I 'm afraid , ' said Alice , timidly ; 'some of the words have got altered . ' 'It is wrong from beginning to end , ' said the Caterpillar decidedly , and there was silence for some minutes . The Caterpillar was the first to speak . 'What size do you want to be ? ' it asked . 'Oh , I 'm not particular as to size , ' Alice hastily replied ; 'only one does n't like changing so often , you know . ' 'I DON'T know , ' said the Caterpillar . Alice said nothing : she had ne
 ver been so much contradicted in her life before , and she felt that she was losing her temper . 'Are you content now ? ' said the Caterpillar . 'Well , I should like to be a LITTLE larger , sir , if you would n't mind , ' said Alice : 'three inches is such a wretched height to be . ' 'It is a very good height indeed ! ' said the Caterpillar angrily , rearing itself upright as it spoke ( it was exactly three inches high ) . 'But I 'm not used to it ! ' pleaded poor Alice in a piteous tone . And she thought of herself , 'I wish the creatures would n't be so easily offended ! ' 'You 'll get used to it in time , ' said the Caterpillar ; and it put the hookah into its mouth and began smoking again . This time Alice waited patiently until it chose to speak again . In a minute or two the Caterpillar took the hookah out of its mouth and yawned once or twice , and shook itself . Then it got down off the mushroom , and crawled away in the grass , merely remarking as it went , 'One side will 
 make you grow taller , and the other side will make you grow shorter . ' 'One side of WHAT ? The other side of WHAT ? ' thought Alice to herself . 'Of the mushroom , ' said the Caterpillar , just as if she had asked it aloud ; and in another moment it was out of sight . Alice remained looking thoughtfully at the mushroom for a minute , trying to make out which were the two sides of it ; and as it was perfectly round , she found this a very difficult question . However , at last she stretched her arms round it as far as they would go , and broke off a bit of the edge with each hand . 'And now which is which ? ' she said to herself , and nibbled a little of the right-hand bit to try the effect : the next moment she felt a violent blow underneath her chin : it had struck her foot ! She was a good deal frightened by this very sudden change , but she felt that there was no time to be lost , as she was shrinking rapidly ; so she set to work at once to eat some of the other bit . Her chin 
 was pressed so closely against her foot , that there was hardly room to open her mouth ; but she did it at last , and managed to swallow a morsel of the lefthand bit . * * * * * * * * * * * * * * * * * * * * 'Come , my head 's free at last ! ' said Alice in a tone of delight , which changed into alarm in another moment , when she found that her shoulders were nowhere to be found : all she could see , when she looked down , was an immense length of neck , which seemed to rise like a stalk out of a sea of green leaves that lay far below her . 'What CAN all that green stuff be ? ' said Alice . 'And where HAVE my shoulders got to ? And oh , my poor hands , how is it I ca n't see you ? ' She was moving them about as she spoke , but no result seemed to follow , except a little shaking among the distant green leaves . As there seemed to be no chance of getting her hands up to her head , she tried to get her head down to them , and was delighted to find that her neck would bend about easily
  in any direction , like a serpent . She had just succeeded in curving it down into a graceful zigzag , and was going to dive in among the leaves , which she found to be nothing but the tops of the trees under which she had been wandering , when a sharp hiss made her draw back in a hurry : a large pigeon had flown into her face , and was beating her violently with its wings . 'Serpent ! ' screamed the Pigeon . 'I 'm NOT a serpent ! ' said Alice indignantly . 'Let me alone ! ' 'Serpent , I say again ! ' repeated the Pigeon , but in a more subdued tone , and added with a kind of sob , 'I 've tried every way , and nothing seems to suit them ! ' 'I have n't the least idea what you 're talking about , ' said Alice . 'I 've tried the roots of trees , and I 've tried banks , and I 've tried hedges , ' the Pigeon went on , without attending to her ; 'but those serpents ! There 's no pleasing them ! ' Alice was more and more puzzled , but she thought there was no use in saying anything more 
 till the Pigeon had finished . 'As if it was n't trouble enough hatching the eggs , ' said the Pigeon ; 'but I must be on the look-out for serpents night and day ! Why , I have n't had a wink of sleep these three weeks ! ' 'I 'm very sorry you 've been annoyed , ' said Alice , who was beginning to see its meaning . 'And just as I 'd taken the highest tree in the wood , ' continued the Pigeon , raising its voice to a shriek , 'and just as I was thinking I should be free of them at last , they must needs come wriggling down from the sky ! Ugh , Serpent ! ' 'But I 'm NOT a serpent , I tell you ! ' said Alice . 'I 'm a--I 'm a-- ' 'Well ! WHAT are you ? ' said the Pigeon . 'I can see you 're trying to invent something ! ' 'I--I 'm a little girl , ' said Alice , rather doubtfully , as she remembered the number of changes she had gone through that day . 'A likely story indeed ! ' said the Pigeon in a tone of the deepest contempt . 'I 've seen a good many little girls in my time , but neve
 r ONE with such a neck as that ! No , no ! You 're a serpent ; and there 's no use denying it . I suppose you 'll be telling me next that you never tasted an egg ! ' 'I HAVE tasted eggs , certainly , ' said Alice , who was a very truthful child ; 'but little girls eat eggs quite as much as serpents do , you know . ' 'I do n't believe it , ' said the Pigeon ; 'but if they do , why then they 're a kind of serpent , that 's all I can say . ' This was such a new idea to Alice , that she was quite silent for a minute or two , which gave the Pigeon the opportunity of adding , 'You 're looking for eggs , I know THAT well enough ; and what does it matter to me whether you 're a little girl or a serpent ? ' 'It matters a good deal to ME , ' said Alice hastily ; 'but I 'm not looking for eggs , as it happens ; and if I was , I should n't want YOURS : I do n't like them raw . ' 'Well , be off , then ! ' said the Pigeon in a sulky tone , as it settled down again into its nest . Alice crouched d
 own among the trees as well as she could , for her neck kept getting entangled among the branches , and every now and then she had to stop and untwist it . After a while she remembered that she still held the pieces of mushroom in her hands , and she set to work very carefully , nibbling first at one and then at the other , and growing sometimes taller and sometimes shorter , until she had succeeded in bringing herself down to her usual height . It was so long since she had been anything near the right size , that it felt quite strange at first ; but she got used to it in a few minutes , and began talking to herself , as usual . 'Come , there 's half my plan done now ! How puzzling all these changes are ! I 'm never sure what I 'm going to be , from one minute to another ! However , I 've got back to my right size : the next thing is , to get into that beautiful garden--how IS that to be done , I wonder ? ' As she said this , she came suddenly upon an open place , with a little hous
 e in it about four feet high . 'Whoever lives there , ' thought Alice , 'it 'll never do to come upon them THIS size : why , I should frighten them out of their wits ! ' So she began nibbling at the righthand bit again , and did not venture to go near the house till she had brought herself down to nine inches high . 
\ No newline at end of file


[03/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/185un_CD_PV1080.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/185un_CD_PV1080.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/185un_CD_PV1080.txt
new file mode 100644
index 0000000..cbec4fe
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/185un_CD_PV1080.txt
@@ -0,0 +1,2 @@
+
+Held at the Palais des Nations , Geneva , on Thursday , 30 August 2007 , at 10.15 a. m. President : Mr. Faysal KHABBAZ HAMOUI ( Syrian Arab Republic ) The PRESIDENT ( spoke in Arabic ) : I declare open the 1080th plenary meeting of the Conference on Disarmament . At the outset , I should like to extend a warm welcome on behalf of the Conference on Disarmament to the participants in the United Nations Programme of Fellowships on Disarmament who are observing the proceedings of this plenary . I am quite sure they will benefit from their participation in this forum , especially from the presentations that will be made to them on the various aspects of the work of the Conference . On behalf of the Conference on Disarmament and on my own behalf , I should like to bid farewell to Ambassador Elisabeth Bonnier , our distinguished colleague from Sweden , who will be leaving Geneva soon to take up other important duties , having brought with her a wealth of experience gained within the Swedis
 h Foreign Ministry and abroad , including a first round from 1983 to 1988 in the Conference on Disarmament and also as a member of the Secretary-General 's Advisory Board on Disarmament Matters . Ambassador Elisabeth Bonnier joined us in 2003 as the Permanent Representative of Sweden to the United Nations Office and other international organizations in Geneva . As a representative of a country with a pre-eminent focus on multilateral disarmament and non-proliferation , she has always been actively and insistently involved in the effort to break the deadlock in this body and to resume the substantive work . I should like to express my particular gratitude to her for her extensive efforts to end the stalemate in this body as one of the six Presidents of this year 's session . On behalf of the Conference and on my own behalf , I would like to convey to Ambassador Bonnier every wish for success and happiness . While we are sad to see Ambassador Bonnier leave Geneva , I am happy to exten
 d a cordial welcome to our new colleague from Canada , Ambassador Marius Grinius . On behalf of the Conference on Disarmament and on my own behalf , I would like to assure Ambassador Grinius of our readiness to provide him with every kind of assistance and support for his work in Geneva . Before turning to the speakers for today , I would like to introduce the Presidential report on the third part of the 2007 session . I closely followed the consultations that the former President of the Conference on Disarmament , Ambassador J�rg Streuli , held on the Presidential proposal . I also followed the work of the six Presidents on the same matter . A series of meetings were held with a large number of Ambassadors of member States . Allow me to summarize the results of these activities as set forth in the Presidential report on part 3 of the 2007 session , which was distributed to you today by the secretariat . My evaluation of the results consists of three points . The first point is tha
 t a wide range of delegations expressed their support for the Presidential proposal based on the three annexed documents , L.1 CRP.5 and CRP.6 . Although some delegations expressed reservations about certain elements of the proposal , they said that they were willing to join the consensus on the proposal . Others expressed the view that more work and consultations were needed to reach a consensus . The second point is that the Conference on Disarmament has achieved substantial progress by conducting important thematic debates on all agenda items . However , it has not yet been able to reach a consensus on a programme of work . As to the third point , momentum has been created to end the long-standing impasse at the Conference on Disarmament and efforts to reach an agreement must be continued . This is the summary of the three points contained in the Presidential report on part 3 of the 2007 session . I should now like to open the floor to any delegations that wish to make a statemen
 t at today 's meeting . I have no speakers on my list . Does any delegation wish to take the floor ? The Ambassador of Sweden . You have the floor . Ms. BORSIIN BONNIER ( Sweden ) : Mr. President , I thank you very much , and initially , let me warmly welcome you on the assumption of the Chair of this august body . Although I will be leaving Geneva , I assure you that my delegation will continue to do its utmost to support you in the important tasks ahead . I also thank you warmly for your initial kind remarks on my person . This is my last plenary meeting , and therefore , in line with the traditions of the CD , I will conclude with a few personal remarks . Mr. President , for altogether nine years , as you noted , I have entered this Council chamber with hope and with expectations . I have seen this room being the anchor of sincere multilateral work to bring about a more secure world . At times the Conference on Disarmament has even been the centre of gravity for the international
  community in Geneva . I have seen it vibrating with energy , bold ideas , creative intellect focused on problem-solving and a determination to make a difference and to turn the tide . Important pieces of international law have been negotiated here ; compliance with vital norms has been discussed and defended with vigour and conviction . But I have also seen the opposite : an anaemic stalemate with delegations resorting to recitals of ceremonious mantras , covering up the traces of their own passivity with useless finger-pointing and blame games , hiding behind the commas of the rules of procedure and shamelessly abusing the consensus rule to abort any attempt to seriously tackle difficult or sensitive issues - in effect taking the whole multilateral approach to disarmament and arms control hostage to their own particular perceptions , preferences or dictates . But still , I continue to enter this room with hope and expectations . The certainties of yesterday are gone . Security per
 ceptions , threats and challenges are in a state of flux . New dangers are emerging , adding to the burden of the old ones . But the debate of old versus new threats is a meaningless one . Unless we try to face both , neither will be dealt with . In the nuclear field we see more brinkmanship than statesmanship . Globalization works to make complex weapon technologies widely available . Military expenses have risen to well beyond a trillion United States dollars . The illusion of an impenetrable defence is taking new forms . Our space assets with their huge scientific and economic potentials are not secure . Cyber-wars are no longer just science fiction , but can disrupt our societies on a massive scale . The overall balance between global security problems solved and those unsolved is , I believe , negative . The uncertainties of tomorrow are , however - and let us not forget - the opportunities of today . Reluctantly or enthusiastically we will have to accept that global problems i
 n a time of galloping globalization cannot be solved unilaterally , bilaterally or even in small circles of like-minded nations . Global security must be built globally . Effective multilateralism is both necessary and possible . It means that the legitimate security concerns of all States must be acknowledged . It means seizing opportunities when they arise . It means reconsidering positions that yield no results . And it means a readiness to give and take . Global security can never be a zero-sum game . Just as the rule of law is a fundamental cornerstone in domestic affairs , so it must also be in international affairs . Without it we will be on the slippery slope towards anarchy and the use of force . Laws are sometimes broken , but this does not make them less essential . Quite the opposite . It reinforces the need to uphold and strengthen them . This is particularly important in the area of international security relations . But there are serious lacunae , both as regards the 
 laws themselves and our ability to see them complied with . We do have a forum where we can approach all these difficult realities together . It is called the Conference on Disarmament . Rejecting the use of the CD is irresponsible . Those blocking the potentials of this body should understand that they are inflicting real damage not only to the security of others , to our common security , but in the long run also to their own . Since the beginning of this year I have had the privilege of working as one of the P-6 . This Presidential sextet has jointly and consistently tried to put the CD back to work , tried to move us from talk shop to workshop , tried to make us take on the challenges of our time . And by now , most delegations stand ready to try . We have come very close to begin a beginning . As I am now leaving , I am sorry not to be with you when you cross the last bridge . Because cross it you will . There is no other way . Now there remains for me only to express my many a
 nd sincere thanks : to the Secretary-General for his support , wisdom and friendship , always at our side . To my fellow P-6 colleagues and our coordinators for their remarkable team spirit and seriousness of purpose . To my colleagues , the CD Ambassadors and their delegations for their many expressions of support and friendship . To everyone in the secretariat who keeps toiling for us in good spirits and solidarity . To the interpreters who try to make sense of our sometimes incomprehensible utterings in all languages . And to the representatives of civil society who with great patience and frustration stay engaged . Thank you all . When I leave this Council chamber today I will do so as I have always entered it - with hope and expectations . The PRESIDENT ( spoke in Arabic ) : I thank the Ambassador of Sweden for her kind wishes addressed to the Conference , and we in turn wish her every success . The delegation of France has asked for the floor . You have the floor , Madam . Ms.
  RAMIS-PLUM ( France ) ( spoke in French ) : I simply wished to raise a point of a technical or even presentational nature in the draft decision which is attached to the report on the third part of our session . I think there is simply a small presentational matter which , in the opinion of my delegation , needs to be rectified . The body of decision L.1 has been separated presentationally from the complementary Presidential statement . We consider that , in the original version of this document these two elements which from a whole were together and should remain together , as they were in the text of decision L.1 which has been distributed until now . The PRESIDENT ( spoke in Arabic ) : I thank the representative of France . We shall discuss this matter with the secretariat . Does any delegation wish to make a statement ? I do not see any . Concerning next week 's work , the secretariat , later this afternoon , will place the draft report on the 2007 session in the members ' pigeo
 nholes . For ease of access , the secretariat will also send out an electronic copy , in PDF format , to the regional coordinators , who can then distribute the document to the members of their respective groups . Next week , i. e. on Tuesday 4 September 2007 , I will proceed with the first reading . As usual , this will be done at an informal meeting right after the formal plenary , which will start at 10 a. m. in this room . Before bringing this morning 's work to a close I wish to make some comments on the final report . First , I should like to thank my colleagues the Ambassadors of the G-6 for their strong support for this draft report . I also wish to thank the secretariat for its outstanding efforts which helped us to complete this draft on time for it to be translated into all the United Nations official languages . As I have already said , the G-6 chairpersons helped us to prepare this report and supported it once it had been completed . The report reflects the consensus in
  the group . We all hope that this consensus will lead to a consensus within the Conference , especially given that the G-6 represents a wide cross section of the Conference membership . We look forward to your cooperation in adopting this report , which we endeavoured to ensure would contain an objective and impartial account of the proceedings at this year 's Conference . We did our utmost to take account of the concerns of all the members , which was not an easy task . The President hopes that when the draft text is discussed next Tuesday , it will enjoy the support of you all , that you will take account of these comments , and show the necessary flexibility for its adoption . Thank you for your understanding . Before closing the meeting , does any delegation wish to take the floor ? I see none . I declare this meeting adjourned . The meeting rose at 10.45 a. m. ( Ms. Borsiin Bonnier , Sweden ) ( The President ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/186un_CD_PV1157.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/186un_CD_PV1157.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/186un_CD_PV1157.txt
new file mode 100644
index 0000000..197c51c
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/186un_CD_PV1157.txt
@@ -0,0 +1,2 @@
+
+Held at the Palais des Nations , Geneva , on Wednesday , 2 September 2009 , at 10.10 a. m. President : Mr. Christian STROHAL ( Austria ) The PRESIDENT : I declare open the 1157th plenary meeting of the Conference on Disarmament . It is a particular honour and pleasure for me to be able to welcome the Federal Minister for European and International Affairs of the Republic of Austria , Mr. Michael Spindelegger , to this Conference . Mr. Spindelegger has had a long career , in spite of his young age , in the Austrian Parliament and the European Parliament . He has been the Second President of the Austrian Parliament and the foreign policy spokesperson of the People 's Party , and became a minister last year . We are delighted that he has made it to Geneva , where he has arrived at a busy time when we are preparing not only for the closure of this year 's session of the Conference on Disarmament but also for the General Assembly of the United Nations , as well as for the relevant activi
 ties of the Security Council , on which Austria will be represented next year as a non-permanent member . I now give the floor to the minister . Mr. SPINDELEGGER ( Austria ) : Mr. President , Thank you for this opportunity to address the Conference on Disarmament today . Austria has assumed the presidency of the Conference at an important point in time . This year the Conference finally adopted a programme of work after more than a decade of stalemate . This is a positive , long overdue development . So I do not need to speak about deadlocks or the missed opportunities of the past . Let me focus on recent positive developments in arms control and disarmament and their implications for our future . The year 2009 has seen one paramount development : the goal of nuclear disarmament and the eventual total elimination of nuclear weapons has been put back on the global agenda . I welcome the five-point plan for disarmament proposed by the Secretary-General of the United Nations , Mr. Ban 
 Ki-moon , which I had the opportunity to discuss with him a few days ago in Austria . I also welcome the recent positive statements that have been made by United States President Obama in Prague and Cairo and by Russian President Medvedev . And only a few weeks ago , Chinese Foreign Minister Yang underlined China 's goal of a nuclear-weapon-free world here in this very room . Austria promoted the vision of a world free of nuclear weapons throughout the cold war , when only a few countries possessed nuclear capabilities . Since proliferation has introduced new threats from States and non-State actors , striving for this goal is now all the more necessary . Our vision is now shared by the most important nuclear powers . But how far has this vision been translated into concrete action ? We can identify two developments . First , direct bilateral negotiations : we see a renewed readiness to engage in bilateral negotiations . The commitment by President Obama and President Medvedev to ne
 gotiate a new strategic arms reduction treaty by the end of this year is a good example . Second , the global atmosphere in multilateral arms control negotiations has changed : preparations for the 2010 Review Conference of the Parties to the Treaty on the Non-Proliferation of Nuclear Weapons ( NPT ) show a markedly better atmosphere . Initiatives by States and civil society all around the world , such as the Security Council summit on nuclear non-proliferation and disarmament due to be held on 24 September , ensure global ownership and demonstrate a new readiness to engage in dialogue . But let me be clear : positive declarations , a good atmosphere and universal interest also need real implementation and compliance . The NPT can only fulfil its promise of peace if all States adhere to their obligations . Similarly the programme of work of the Conference on Disarmament needs to be implemented if it is to become a real achievement . This brings me to my next point - our expectations
  for the future . First let me start with the Comprehensive Nuclear-Test-Ban Treaty ( CTBT ) , a treaty which originated in this chamber . The deplorable nuclear test conducted by the Democratic People 's Republic of Korea in May has made clear that the entry into force of the CTBT is overdue . As Co-Chair , together with Costa Rica , Austria has been deeply involved in efforts to secure this goal over the past two years . Let me thus use this opportunity to appeal once again to those States that have not yet done so to accede to the CTBT as soon as possible . I am encouraged by the recent commitment of President Obama to pursue ratification by the United States , and also by China 's comments in this regard . Let me appeal to all States to show their support for the treaty at the Article XIV conference in New York later this month , and to all delegation 's present to help ensure that their country participates . Second , Austria believes that a comprehensive fissile material cut-o
 ff treaty ( FMCT ) - a treaty that still has to emerge from this chamber - would significantly contribute to nuclear disarmament . Reliable techniques for monitoring and verification have been developed over the past years . The potential positive impact of such a treaty has been confirmed many times . The IAEA is ready to make an important contribution . It now requires the political will of Governments and the efforts of skilled negotiators to make it a reality . It is my sincere hope that the new spirit in the Conference on Disarmament will soon enable it to proceed with actual work and start negotiations on an FMCT . Finally , when we talk about expectations for the upcoming year , we must mention the NPT Review Conference , which will take place next May in New York . The Preparatory Committee has shown a willingness to achieve a meaningful outcome this time . Today , we see a broad understanding that the dangers of nuclear technology need to be contained in a manner that creat
 es trust and confidence among all States . The Review Conference will demonstrate whether we are capable of transforming this positive atmosphere into real progress . Austria will present and support initiatives aimed at making substantial progress in nuclear disarmament , non-proliferation and a global system to effectively control the nuclear fuel cycle . This should help to avoid tensions such as the ones we are currently witnessing in regard to the nuclear programme of the Islamic Republic of Iran . Austria attaches great importance to arms control also in the area of conventional weapons such as landmines , cluster munitions and small arms and light weapons . Let me make three brief comments in this respect . First , with regard to landmines : Austria 's objective for the Second Review Conference of the Anti-Personnel Mine Ban Convention , to be held later this year in Cartagena , Columbia , is to work for a strong Cartagena action plan , which will serve as a clear guide to St
 ates and bring us closer to a world free of mines . One priority for us is to improve assistance to landmine survivors . In this respect Austria has sponsored - together with Norway and civil society partners - a project to examine the global impact of the Anti-Personnel Mine Ban Convention and what remains to be done to improve the situation of survivors . The report will be presented here today right after this meeting . Let me just use this opportunity to encourage all States that have not yet done so to accede to the Anti-Personnel Mine Ban Convention . It is time for the international community not only to review its stance on nuclear weapons but also to finally eradicate this terrible weapon . Second , with regard to cluster munitions : I encourage States to ratify the Convention on Cluster Munitions . So far 17 States have ratified it . It is our strong hope that the Convention will receive the necessary 30 ratifications to enter into force before the end of this year . To su
 pport this , Austria has engaged in a worldwide campaign and will support a conference on cluster munitions in Indonesia in November . Third , with regard to small arms and light weapons : these are a major security threat in parts of the world . Former United Nations Secretary-General Kofi Annan once called them the \u201c real weapons of mass destruction \u201d . Every year , hundreds of thousands of people are killed by them . The United Nations Programme of Action to Prevent , Combat and Eradicate the Illicit Trade in Small Arms and Light Weapons in All Its Aspects is a significant step forward . Austria supports its implementation on the ground , especially in Africa . To stop the illicit arms trade , Austria supports the conclusion of a legally-binding international arms trade treaty . As a producing and exporting country of small arms , Austria feels a special responsibility to engage in this process and will also host the concluding conference of the global joint project of the Eu
 ropean Union and the United Nations Institute for Disarmament Research in February 2010. We hope that the conference in Vienna will provide positive momentum and enable the conclusion of a treaty in the near future . One thing is clear : there is no lack of work in the field of arms control and disarmament today and the international community is responding positively . In order to succeed , I count not only on the new flexibility of the nuclear-weapon States and the constructive contributions of all the other States , but also on the commitment and expertise of civil society . Throughout the years , non-governmental organizations and private citizens have remained important motors of the disarmament and arms control process . Austria has always underlined that disarmament and arms control must be an open , participatory process if it is to succeed . Protecting the world from the harm caused by weapons is a global project and needs active support from all of us . In this respect , I
  would like to thank the Secretary-General of the Conference for his tireless efforts , and this year 's other presidencies - Viet Nam , Zimbabwe , Algeria , Argentina and Australia - for their excellent cooperation in the Conference on Disarmament . I wish the Conference the necessary energy for a good conclusion to this year 's session and an early and effective resumption next year . The PRESIDENT : I thank the minister for his statement . Before adjourning this plenary meeting , let me just make two brief announcements . The first is , as mentioned , at 11 a. m. in this chamber , the minister will launch the report \u201c Voices from the Ground \u201d , a report in which victims of landmines and explosive remnants of war will speak out on victim assistance . The next plenary meeting of the Conference will be held tomorrow at 11 a. m. in this chamber . At that time , the Chairman-Designate of the First Committee of the General Assembly , Ambassador Jos� Luis Cancela of Uruguay , will 
 deliver a statement . This plenary meeting is adjourned . The meeting rose at 10.25 a. m. ( Mr. Spindelegger , ( Austria ) ( Mr. Spindelegger , ( Austria ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/187un_CD_PV857.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/187un_CD_PV857.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/187un_CD_PV857.txt
new file mode 100644
index 0000000..904d93a
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/187un_CD_PV857.txt
@@ -0,0 +1,2 @@
+
+24 August 2000 Original : ENGLISH Held at the Palais des Nations , Geneva , on Thursday , 24 August 2000 , at 10.10 a. m. President : Mr. Celso Amorim ( Brazil ) The PRESIDENT : I declare open the 857th plenary meeting of the Conference . Distinguished delegates , we have learned with deep sorrow the tragic death of the crew members of the Russian nuclear submarine Kursk . I am sure that you all join me in extending our deep sympathy and condolences to the delegation of the Russian Federation on this sad occasion . I would kindly request our Russian colleague to convey these sentiments to his Government and to the bereaved families , as well as to the whole Russian people . I should like to extend a warm welcome , on behalf of all of us , to the new representative of Romania , Ambassador Anda-Cristina Filip and assure her of our full cooperation and support . There is no speaker inscribed on my list for today . Does any delegation wish to take the floor at this stage ? Japan , you h
 ave the floor , sir . Mr. NORBURU ( Japan ) : With your indulgence , Mr. President , I should like to take this opportunity to mention briefly how much our delegation appreciates all the efforts being made by our Brazilian President to reach an agreement on the programme of work for the Conference on Disarmament . At the same time , we must also express our sincere recognition of his predecessors , whose efforts were also most worthy of note . Ambassador Amorim has already eloquently demonstrated how far a President can push all delegations forward in seeking a solution to the apparently impossible problem faced by the Conference on Disarmament . Ambassador Amorim has been conducting this task with perseverance , wisdom and diplomatic skills which my delegation highly esteems , and his work will stand as a textbook case of good chairmanship . What Ambassador Amorim has been able to achieve so far is remarkable , and we must all be very grateful that he intends to continue working un
 til the last moment . Mr. President , it may be easy for people like us who are constantly exposed to the difficulties of international relations to understand why we have not yet been able to make progress in the Conference on Disarmament . This may not , however , hold true for the rest of society outside our diplomatic realm , who are diligently watching us and expecting real outcomes from us . This is reason enough for us not to abandon the possibility of agreeing on a programme of work , even at this late stage of this year 's session of the Conference on Disarmament . We must carry on with the work this week and during the remaining four weeks of this session . Mr. President , many delegations , including mine , consider extremely important all the efforts being made to accomplish progress in nuclear disarmament as the pivotal foreign policy objective , be it FMCT or a general discussion on nuclear disarmament . Over the last two years we have experienced an uncomfortable ambi
 ence of uncertainty in the Conference on Disarmament , but my delegation takes some solace from the fact that under you , Mr. President , we are still trying to be upbeat in our efforts to bring an end to this troubling reality . The PRESIDENT : I thank very much the representative of Japan . I am really moved by his words and encouraged to try to come to an agreement in this which is the last leg of my presidency . Is there any other delegation which wants to take the floor ? That does not seem to be the case . As you are aware , during the last few days , I have intensified consultations aimed at promoting consensus on the Programme of Work of the Conference on the basis of the ideas which I presented to all of you . I am encouraged by the fact that all delegations considered those ideas as a basis for further intensified consultations . I take the point made yesterday by several distinguished ambassadors that , at this stage , a basis for further intensified consultations is not 
 good enough . I shall revert to this item later . I convened informal open-ended consultations yesterday with the purpose of ascertaining whether further progress can be made on a consensus decision on the Programme of Work . I was pleased to see that there was wide support from a broad cross-section of nations from different regions , different geopolitical situations , to my efforts , and I was also pleased to see the readiness displayed by all delegations to work towards compromise . Therefore , as I announced yesterday in the presidential consultations , it is my intention to suspend this plenary meeting and to convene in 10 minutes an informal open-ended consultation to discuss the ideas I have been working on . But before doing so , if you allow me a lighter note , I will mention to you that a friend of mine , who is actually our Ambassador in South Africa , recently gave me an old book called \u201c Legends of the United Nations \u201d . It was a collection by I believe a British a
 uthor , Frances Frost , of legends of almost all the nations that were then comprising the United Nations written just after the Charter was approved in San Francisco . One of the legends that came across to me , which I think is especially significant to what we are living today ( and I will not mention the country from where the legend is in order not to be misinterpreted in my purpose in mentioning the legend ) . It goes more or less as follows : Once upon a time there was a king and a princess , and the king wanted the princess - a very beautiful princess - to get married , but the condition was that the suitor should bring a blue rose . Well , a blue rose does not exist . So there were several suitors , merchants , warriors and others , and they tried different solutions . One brought a beautiful crystal ball with a blue rose inscribed in it . Another one tried a silk rose . Another one tried just to dye , to use a blue dye in a white rose , so that also he could be accepted . 
 But of course none of these tricks was accepted . But in the end there was another one who came with a white rose . He just plucked a white rose and brought it to the palace . The king and the princess considered the suitor and thought that he was a worthy gentleman and that the marriage could be a good one , so the king and the princess , to the astonishment of all those who were present , said : \u201c Well , here we have a blue rose \u201d . And they lived happily ever after . The meeting was suspended at 10.20 a. m. and resumed at 12.15 p. m. The PRESIDENT : We resume our formal meeting . We have just failed in approving a Programme of Work for the CD . There is nothing extraordinary about this . This has actually been the pattern for several months and years . Failure is apparently now part of our routine . We all have to reflect on the causes of it and how to overcome it . I do not think this is the appropriate time to draw lessons . Certainly , in my case , any analysis that I woul
 d try now would be tinged with a sense of personal frustration and emotions are not good counsellors for rational thinking . I may come back to that sometime in the future from the bench of Brazil . I do not need to explain either the gist of my efforts , how I tried to explore ambiguities in a positive way , how I tried to test the limits of the possible . My text will speak for itself , with its possible merits and obvious shortcomings . I am asking the secretariat to circulate it as a CD document , not because I am under any illusion that what was considered as a basis in the last days will still be a basis in four or five months ' time , but I thought in any case it would be useful to have that text as part of our records to show the general direction of the efforts we made collectively . At least it will demonstrate we did not give up easily . As to the future of the CD , it is under great doubt . The CD is a tool , and a tool that is not used gets rusted . Will it be a tragedy
  if , as some have already even mentioned here , the CD disappears ? I honestly do not know . Whatever will be deemed collectively indispensable by those who detain the power of destruction and annihilation will be done bilaterally , trilaterally , plurilaterally . But for those of us who rely on multilateralism , on the power of rational persuasion , and do not dispose in the same proportion - either because they cannot or do not want to have the force of arms , actual or potential - there will be a great loss . The system of world governance , if I may use that word in a vital area such as security , will be tremendously impoverished . I hope it is not too late to prevent that from happening . There is a positive side to those efforts that we all undertook , despite the fact that they did not succeed : the way we worked together , trying to look for reasons to be optimists in the midst of so many factors is encouraging . Clearly - and this was the case yesterday and today in the i
 nformal meetings - I received from a cross-section of the membership great support not only for my efforts but also for my proposal as a basis for a consensus . And even from the others , who could not at least explicitly join these expressions , I received loyal , faithful support , and I am glad that they were able to agree that my non-paper was a basis for further consultations . This engagement , which existed all along , shows that cynicism and unreasoned pessimism have not , or at least not yet , taken over the CD . I would like in this connection to quote from Amartya Sen , the famous Nobel Prize winner 's recent essay in the New York Review of Books : \u201c Unreasoned pessimism masquerading as composure based on realism and common sense can serve to ` justify ' disastrous inaction and an abdication of public responsibility \u201d . Let us not succumb to it . Finally , I wish to express once again my sincere appreciation for the invaluable support I received from Group Coordinator
 s , as well as all other colleagues and their respective delegations , through these actually two months of presidency . I am also grateful to the Secretary-General of the CD , Mr. Petrovsky , and his deputy , Mr. Bensmail , for the way in which ( The President ) they put at my disposal their vast experience and expertise . I also want to thank the staff of the CD , as well as the interpreters . Finally , I am deeply indebted to my collaborators for their hard work and advice and , more importantly , the optimism they shared with me in the course of the Brazilian presidency of the CD . The Russian Federation has asked for the floor . Mr. SIDOROV ( Russian Federation ) ( translated from Russian ) : Thank you , Mr. President . I have asked for the floor to thank you for your words of condolence on the occasion of the tragedy that struck the Russian nuclear submarine Kursk . Your condolences will be conveyed to the Russian Government and to the families of the crew members . The PRESID
 ENT : Thank you . No other requests for the floor ? The next plenary meeting of the Conference will be held on Thursday , 31 August 2000 , at 10 a. m. The meeting rose at 12.25 p. m. 
\ No newline at end of file


[26/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Marx/78MarxMarxK_JewishQ_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Marx/78MarxMarxK_JewishQ_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Marx/78MarxMarxK_JewishQ_EN.txt.txt
new file mode 100644
index 0000000..e499164
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Marx/78MarxMarxK_JewishQ_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff I Bruno Bauer , The Jewish Question , Braunschweig , 1843 The German Jews desire emancipation . What kind of emancipation do they desire ? Civic , political emancipation . Bruno Bauer replies to them : No one in Germany is politically emancipated . We ourselves are not free . How are we to free you ? You Jews are egoists if you demand a special emancipation for yourselves as Jews . As Germans , you ought to work for the political emancipation of Germany , and as human beings , for the emancipation of mankind , and you should feel the particular kind of your oppression and your shame not as an exception to the rule , but on the contrary as a confirmation of the rule . Or do the Jews demand the same status as Christian subjects of the state ? In that case , they recognize that the Christian state is justified and they recognize , too , the regime of general oppression . Why should they disapprove of their special yoke if they approve of the general yoke ? Why should the German be 
 interested in the liberation of the Jew , if the Jew is not interested in the liberation of the German ? The Christian state knows only privileges . In this state , the Jew has the privilege of being a Jew . As a Jew , he has rights which the Christians do not have . Why should he want rights which he does not have , but which the Christians enjoy ? In wanting to be emancipated from the Christian state , the Jew is demanding that the Christian state should give up its religious prejudice . Does he , the Jew , give up his religious prejudice ? Has he , then , the right to demand that someone else should renounce his religion ? By its very nature , the Christian state is incapable of emancipating the Jew ; but , adds Bauer , by his very nature the Jew cannot be emancipated . So long as the state is Christian and the Jew is Jewish , the one is as incapable of granting emancipation as the other is of receiving it . The Christian state can behave towards the Jew only in the way character
 istic of the Christian state \u2013 that is , by granting privileges , by permitting the separation of the Jew from the other subjects , but making him feel the pressure of all the other separate spheres of society , and feel it all the more intensely because he is in religious opposition to the dominant religion . But the Jew , too , can behave towards the state only in a Jewish way \u2013 that is , by treating it as something alien to him , by counterposing his imaginary nationality to the real nationality , by counterposing his illusory law to the real law , by deeming himself justified in separating himself from mankind , by abstaining on principle from taking part in the historical movement , by putting his trust in a future which has nothing in common with the future of mankind in general , and by seeing himself as a member of the Jewish people , and the Jewish people as the chosen people . On what grounds , then , do you Jews want emancipation ? On account of your religion ? It is 
 the mortal enemy of the state religion . As citizens ? In Germany , there are no citizens . As human beings ? But you are no more human beings than those to whom you appeal . Bauer has posed the question of Jewish emancipation in a new form , after giving a critical analysis of the previous formulations and solutions of the question . What , he asks , is the nature of the Jew who is to be emancipated and of the Christian state that is to emancipate him ? He replies by a critique of the Jewish religion , he analyzes the religious opposition between Judaism and Christianity , he elucidates the essence of the Christian state \u2013 and he does all this audaciously , trenchantly , wittily , and with profundity , in a style of writing that is as precise as it is pithy and vigorous . How , then , does Bauer solve the Jewish question ? What is the result ? The formulation of a question is its solution . The critique of the Jewish question is the answer to the Jewish question . The summary , t
 herefore , is as follows : We must emancipate ourselves before we can emancipate others . The most rigid form of the opposition between the Jew and the Christian is the religious opposition . How is an opposition resolved ? By making it impossible . How is religious opposition made impossible ? By abolishing religion . As soon as Jew and Christian recognize that their respective religions are no more than different stages in the development of the human mind , different snake skins cast off by history , and that man is the snake who sloughed them , the relation of Jew and Christian is no longer religious but is only a critical , scientific , and human relation . Science , then , constitutes their unity . But , contradictions in science are resolved by science itself . The German Jew , in particular , is confronted by the general absence of political emancipation and the strongly marked Christian character of the state . In Bauer\u2019s conception , however , the Jewish question has a u
 niversal significance , independent of specifically German conditions . It is the question of the relation of religion to the state , of the contradiction between religious constraint and political emancipation . Emancipation from religion is laid down as a condition , both to the Jew who wants to be emancipated politically , and to the state which is to effect emancipation and is itself to be emancipated . \u201c Very well , \u201d it is said , and the Jew himself says it , \u201c the Jew is to become emancipated not as a Jew , not because he is a Jew , not because he possesses such an excellent , universally human principle of morality ; on the contrary , the Jew will retreat behind the citizen and be a citizen , although he is a Jew and is to remain a Jew . That is to say , he is and remains a Jew , although he is a citizen and lives in universally human conditions : his Jewish and restricted nature triumphs always in the end over his human and political obligations . The prejudice remain
 s in spite of being outstripped by general principles . But if it remains , then , on the contrary , it outstrips everything else . \u201d \u201c Only sophistically , only apparently , would the Jew be able to remain a Jew in the life of the state . Hence , if he wanted to remain a Jew , the mere appearance would become the essential and would triumph ; that is to say , his life in the state would be only a semblance or only a temporary exception to the essential and the rule . \u201d ( \u201c The Capacity of Present-Day Jews and Christians to Become Free , \u201d Einundzwanzig Bogen , pp . 57 ) Let us hear , on the other hand , how Bauer presents the task of the state . \u201c France , \u201d he says , \u201c has recently shown us \u201d ( Proceedings of the Chamber of Deputies , December 26 , 1840 ) \u201c in the connection with the Jewish question \u2013 just as it has continually done in all other political questions \u2013 the spectacle of a life which is free , but which revokes its freedom by law , hence declar
 ing it to be an appearance , and on the other hand contradicting its free laws by its action . \u201d ( The Jewish Question , p. 64 ) \u201c In France , universal freedom is not yet the law , the Jewish question too has not yet been solved , because legal freedom \u2013 the fact that all citizens are equal \u2013 is restricted in actual life , which is still dominated and divided by religious privileges , and this lack of freedom in actual life reacts on law and compels the latter to sanction the division of the citizens , who as such are free , into oppressed and oppressors . \u201d ( p. 65 ) When , therefore , would the Jewish question be solved for France ? \u201c The Jew , for example , would have ceased to be a Jew if he did not allow himself to be prevented by his laws from fulfilling his duty to the state and his fellow citizens , that is , for example , if on the Sabbath he attended the Chamber of Deputies and took part in the official proceedings . Every religious privilege , and therefore a
 lso the monopoly of a privileged church , would have been abolished altogether , and if some or many persons , or even the overwhelming majority , still believed themselves bound to fulfil religious duties , this fulfilment ought to be left to them as a purely private matter . \u201d ( p. 65 ) \u201c There is no longer any religion when there is no longer any privileged religion . Take from religion its exclusive power and it will no longer exist . \u201d ( p. 66 ) \u201c Just as M. Martin du Nord saw the proposal to omit mention of Sunday in the law as a motion to declare that Christianity has ceased to exist , with equal reason ( and this reason is very well founded ) the declaration that the law of the Sabbath is no longer binding on the Jew would be a proclamation abolishing Judaism . \u201d ( p. 71 ) Bauer , therefore , demands , on the one hand , that the Jew should renounce Judaism , and that mankind in general should renounce religion , in order to achieve civic emancipation . On the other
  hand , he quite consistently regards the political abolition of religion as the abolition of religion as such . The state which presupposes religion is not yet a true , real state . \u201c Of course , the religious notion affords security to the state . But to what state ? To what kind of state ? \u201d ( p. 97 ) At this point , the one-sided formulation of the Jewish question becomes evident . It was by no means sufficient to investigate : Who is to emancipate ? Who is to be emancipated ? Criticism had to investigate a third point . It had to inquire : What kind of emancipation is in question ? What conditions follow from the very nature of the emancipation that is demanded ? Only the criticism of political emancipation itself would have been the conclusive criticism of the Jewish question and its real merging in the \u201c general question of time . \u201d Because Bauer does not raise the question to this level , he becomes entangled in contradictions . He puts forward conditions which are n
 ot based on the nature of political emancipation itself . He raises questions which are not part of his problem , and he solves problems which leave this question unanswered . When Bauer says of the opponents of Jewish emancipation : \u201c Their error was only that they assumed the Christian state to be the only true one and did not subject it to the same criticism that they applied to Judaism \u201d ( op . cit . , p. 3 ) , we find that his error lies in the fact that he subjects to criticism only the \u201c Christian state , \u201d not the \u201c state as such , \u201d that he does not investigate the relation of political emancipation to human emancipation and , therefore , puts forward conditions which can be explained only by uncritical confusion of political emancipation with general human emancipation . If Bauer asks the Jews : Have you , from your standpoint , the right to want political emancipation ? We ask the converse question : Does the standpoint of political emancipation give the right
  to demand from the Jew the abolition of Judaism and from man the abolition of religion ? The Jewish question acquires a different form depending on the state in which the Jew lives . In Germany , where there is no political state , no state as such , the Jewish question is a purely theological one . The Jew finds himself in religious opposition to the state , which recognizes Christianity as its basis . This state is a theologian ex professo . Criticism here is criticism of theology , a double-edged criticism \u2013 criticism of Christian theology and of Jewish theology . Hence , we continue to operate in the sphere of theology , however much we may operate critically within it . In France , a constitutional state , the Jewish question is a question of constitutionalism , the question of the incompleteness of political emancipation . Since the semblance of a state religion is retained here , although in a meaningless and self-contradictory formula , that of a religion of the majority 
 , the relation of the Jew to the state retains the semblance of a religious , theological opposition . Only in the North American states \u2013 at least , in some of them \u2013 does the Jewish question lose its theological significance and become a really secular question . Only where the political state exists in its completely developed form can the relation of the Jew , and of the religious man in general , to the political state , and therefore the relation of religion to the state , show itself in its specific character , in its purity . The criticism of this relation ceases to be theological criticism as soon as the state ceases to adopt a theological attitude toward religion , as soon as it behaves towards religion as a state \u2013 i. e. , politically . Criticism , then , becomes criticism of the political state . At this point , where the question ceases to be theological , Bauer\u2019s criticism ceases to be critical . \u201c In the United States there is neither a state religion nor a 
 religion declared to be that of the majority , nor the predominance of one cult over another . The state stands aloof from all cults . \u201d ( Marie ou l\u2019esclavage aux Etats-Unis , etc. , by G. de Beaumont , Paris , 1835 , p. 214 ) Indeed , there are some North American states where \u201c the constitution does not impose any religious belief or religious practice as a condition of political rights . \u201d ( op . cit . , p. 225 ) Nevertheless , \u201c in the United States people do not believe that a man without religion could be an honest man . \u201d ( op . cit . , p. 224 ) Nevertheless , North America is pre-eminently the country of religiosity , as Beaumont , Tocqueville , and the Englishman Hamilton unanimously assure us . The North American states , however , serve us only as an example . The question is : What is the relation of complete political emancipation to religion ? If we find that even in the country of complete political emancipation , religion not only exists , but displays a
  fresh and vigorous vitality , that is proof that the existence of religion is not in contradiction to the perfection of the state . Since , however , the existence of religion is the existence of defect , the source of this defect can only be sought in the nature of the state itself . We no longer regard religion as the cause , but only as the manifestation of secular narrowness . Therefore , we explain the religious limitations of the free citizen by their secular limitations . We do not assert that they must overcome their religious narrowness in order to get rid of their secular restrictions , we assert that they will overcome their religious narrowness once they get rid of their secular restrictions . We do not turn secular questions into theological ones . History has long enough been merged in superstition , we now merge superstition in history . The question of the relation of political emancipation to religion becomes for us the question of the relation of political emancip
 ation to human emancipation . We criticize the religious weakness of the political state by criticizing the political state in its secular form , apart from its weaknesses as regards religion . The contradiction between the state and a particular religion , for instance Judaism , is given by us a human form as the contradiction between the state and particular secular elements ; the contradiction between the state and religion in general as the contradiction between the state and its presuppositions in general . The political emancipation of the Jew , the Christian , and , in general , of religious man , is the emancipation of the state from Judaism , from Christianity , from religion in general . In its own form , in the manner characteristic of its nature , the state as a state emancipates itself from religion by emancipating itself from the state religion \u2013 that is to say , by the state as a state not professing any religion , but , on the contrary , asserting itself as a state
  . The political emancipation from religion is not a religious emancipation that has been carried through to completion and is free from contradiction , because political emancipation is not a form of human emancipation which has been carried through to completion and is free from contradiction . The limits of political emancipation are evident at once from the fact that the state can free itself from a restriction without man being really free from this restriction , that the state can be a free state [ pun on word Freistaat , which also means republic ] without man being a free man . Bauer himself tacitly admits this when he lays down the following condition for political emancipation : \u201c Every religious privilege , and therefore also the monopoly of a privileged church , would have been abolished altogether , and if some or many persons , or even the overwhelming majority , still believed themselves bound to fulfil religious duties , this fulfilment ought to be left to them as 
 a purely private matter . \u201d [ The Jewish Question , p. 65 ] It is possible , therefore , for the state to have emancipated itself from religion even if the overwhelming majority is still religious . And the overwhelming majority does not cease to be religious through being religious in private . But , the attitude of the state , and of the republic [ free state ] in particular , to religion is , after all , only the attitude to religion of the men who compose the state . It follows from this that man frees himself through the medium of the state , that he frees himself politically from a limitation when , in contradiction with himself , he raises himself above this limitation in an abstract , limited , and partial way . It follows further that , by freeing himself politically , man frees himself in a roundabout way , through an intermediary , although an essential intermediary . It follows , finally , that man , even if he proclaims himself an atheist through the medium of the sta
 te \u2013 that is , if he proclaims the state to be atheist \u2013 still remains in the grip of religion , precisely because he acknowledges himself only by a roundabout route , only through an intermediary . Religion is precisely the recognition of man in a roundabout way , through an intermediary . The state is the intermediary between man and man\u2019s freedom . Just as Christ is the intermediary to whom man transfers the burden of all his divinity , all his religious constraint , so the state is the intermediary to whom man transfers all his non-divinity and all his human unconstraint . The political elevation of man above religion shares all the defects and all the advantages of political elevation in general . The state as a state annuls , for instance , private property , man declares by political means that private property is abolished as soon as the property qualification for the right to elect or be elected is abolished , as has occurred in many states of North America . Hamilton
  quite correctly interprets this fact from a political point of view as meaning : \u201c the masses have won a victory over the property owners and financial wealth . \u201d [ Thomas Hamilton , Men and Manners in America , 2 vols , Edinburgh , 1833 , p. 146 ] Is not private property abolished in idea if the non-property owner has become the legislator for the property owner ? The property qualification for the suffrage is the last political form of giving recognition to private property . Nevertheless , the political annulment of private property not only fails to abolish private property but even presupposes it . The state abolishes , in its own way , distinctions of birth , social rank , education , occupation , when it declares that birth , social rank , education , occupation , are non-political distinctions , when it proclaims , without regard to these distinction , that every member of the nation is an equal participant in national sovereignty , when it treats all elements of the re
 al life of the nation from the standpoint of the state . Nevertheless , the state allows private property , education , occupation , to act in their way \u2013 i. e. , as private property , as education , as occupation , and to exert the influence of their special nature . Far from abolishing these real distinctions , the state only exists on the presupposition of their existence ; it feels itself to be a political state and asserts its universality only in opposition to these elements of its being . Hegel , therefore , defines the relation of the political state to religion quite correctly when he says : \u201c In order [ ... ] that the state should come into existence as the self-knowing , moral reality of the mind , its distraction from the form of authority and faith is essential . But this distinction emerges only insofar as the ecclesiastical aspect arrives at a separation within itself . It is only in this way that the state , above the particular churches , has achieved and brough
 t into existence universality of thought , which is the principle of its form \u201d ( Hegel\u2019s Philosophy of Right , 1st edition , p. 346 ) . Of course ! Only in this way , above the particular elements , does the state constitute itself as universality . The perfect political state is , by its nature , man\u2019s species-life , as opposed to his material life . All the preconditions of this egoistic life continue to exist in civil society outside the sphere of the state , but as qualities of civil society . Where the political state has attained its true development , man \u2013 not only in thought , in consciousness , but in reality , in life \u2013 leads a twofold life , a heavenly and an earthly life : life in the political community , in which he considers himself a communal being , and life in civil society , in which he acts as a private individual , regards other men as a means , degrades himself into a means , and becomes the plaything of alien powers . The relation of the political 
 state to civil society is just as spiritual as the relations of heaven to earth . The political state stands in the same opposition to civil society , and it prevails over the latter in the same way as religion prevails over the narrowness of the secular world \u2013 i. e. , by likewise having always to acknowledge it , to restore it , and allow itself to be dominated by it . In his most immediate reality , in civil society , man is a secular being . Here , where he regards himself as a real individual , and is so regarded by others , he is a fictitious phenomenon . In the state , on the other hand , where man is regarded as a species-being , he is the imaginary member of an illusory sovereignty , is deprived of his real individual life and endowed with an unreal universality . Man , as the adherent of a particular religion , finds himself in conflict with his citizenship and with other men as members of the community . This conflict reduces itself to the secular division between the p
 olitical state and civil society . For man as a bourgeois [ i. e. , as a member of civil society , \u201c bourgeois society \u201d in German ] , \u201c life in the state \u201d is \u201c only a semblance or a temporary exception to the essential and the rule . \u201d Of course , the bourgeois , like the Jew , remains only sophistically in the sphere of political life , just as the citoyen [ \u2018 citizen \u2019 in French , i. e. , the participant in political life ] only sophistically remains a Jew or a bourgeois . But , this sophistry is not personal . It is the sophistry of the political state itself . The difference between the merchant and the citizen [ Staatsb�rger ] , between the day-laborer and the citizen , between the landowner and the citizen , between the merchant and the citizen , between the living individual and the citizen . The contradiction in which the religious man finds himself with the political man is the same contradiction in which the bourgeois finds himself with the citoyen , and
  the member of civil society with his political lion\u2019s skin . This secular conflict , to which the Jewish question ultimately reduces itself , the relation between the political state and its preconditions , whether these are material elements , such as private property , etc. , or spiritual elements , such as culture or religion , the conflict between the general interest and private interest , the schism between the political state and civil society \u2013 these secular antitheses Bauer allows to persist , whereas he conducts a polemic against their religious expression . \u201c It is precisely the basis of civil society , the need that ensures the continuance of this society and guarantees its necessity , which exposes its existence to continual dangers , maintains in it an element of uncertainty , and produces that continually changing mixture of poverty and riches , of distress and prosperity , and brings about change in general . \u201d ( p. 8 ) Compare the whole section : \u201c Civil 
 Society \u201d ( pp . 8-9 ) , which has been drawn up along the basic lines of Hegel\u2019s philosophy of law . Civil society , in its opposition to the political state , is recognized as necessary , because the political state is recognized as necessary . Political emancipation is , of course , a big step forward . True , it is not the final form of human emancipation in general , but it is the final form of human emancipation within the hitherto existing world order . It goes without saying that we are speaking here of real , practical emancipation . Man emancipates himself politically from religion by banishing it from the sphere of public law to that of private law . Religion is no longer the spirit of the state , in which man behaves \u2013 although in a limited way , in a particular form , and in a particular sphere \u2013 as a species-being , in community with other men . Religion has become the spirit of civil society , of the sphere of egoism , of bellum omnium contra omnes . It is no 
 longer the essence of community , but the essence of difference . It has become the expression of man\u2019s separation from his community , from himself and from other men \u2013 as it was originally . It is only the abstract avowal of specific perversity , private whimsy , and arbitrariness . The endless fragmentation of religion in North America , for example , gives it even externally the form of a purely individual affair . It has been thrust among the multitude of private interests and ejected from the community as such . But one should be under no illusion about the limits of political emancipation . The division of the human being into a public man and a private man , the displacement of religion from the state into civil society , this is not a stage of political emancipation but its completion ; this emancipation , therefore , neither abolished the real religiousness of man , nor strives to do so . The decomposition of man into Jew and citizen , Protestant and citizen , religiou
 s man and citizen , is neither a deception directed against citizenhood , nor is it a circumvention of political emancipation , it is political emancipation itself , the political method of emancipating oneself from religion . Of course , in periods when the political state as such is born violently out of civil society , when political liberation is the form in which men strive to achieve their liberation , the state can and must go as far as the abolition of religion , the destruction of religion . But it can do so only in the same way that it proceeds to the abolition of private property , to the maximum , to confiscation , to progressive taxation , just as it goes as far as the abolition of life , the guillotine . At times of special self-confidence , political life seeks to suppress its prerequisite , civil society and the elements composing this society , and to constitute itself as the real species-life of man , devoid of contradictions . But , it can achieve this only by com
 ing into violent contradiction with its own conditions of life , only by declaring the revolution to be permanent , and , therefore , the political drama necessarily ends with the re-establishment of religion , private property , and all elements of civil society , just as war ends with peace . Indeed , the perfect Christian state is not the so-called Christian state \u2013 which acknowledges Christianity as its basis , as the state religion , and , therefore , adopts an exclusive attitude towards other religions . On the contrary , the perfect Christian state is the atheistic state , the democratic state , the state which relegates religion to a place among the other elements of civil society . The state which is still theological , which still officially professes Christianity as its creed , which still does not dare to proclaim itself as a state , has , in its reality as a state , not yet succeeded in expressing the human basis \u2013 of which Christianity is the high-flown expression 
 \u2013 in a secular , human form . The so-called Christian state is simply nothing more than a non-state , since it is not Christianity as a religion , but only the human background of the Christian religion , which can find its expression in actual human creations . The so-called Christian state is the Christian negation of the state , but by no means the political realization of Christianity . The state which still professes Christianity in the form of religion , does not yet profess it in the form appropriate to the state , for it still has a religious attitude towards religion \u2013 that is to say , it is not the true implementation of the human basis of religion , because it still relies on the unreal , imaginary form of this human core . The so-called Christian state is the imperfect state , and the Christian religion is regarded by it as the supplementation and sanctification of its imperfection . For the Christian state , therefore , religion necessarily becomes a means ; hence ,
  it is a hypocritical state . It makes a great difference whether the complete state , because of the defect inherent in the general nature of the state , counts religion among its presuppositions , or whether the incomplete state , because of the defect inherent in its particular existence as a defective state , declares that religion is its basis . In the latter case , religion becomes imperfect politics . In the former case , the imperfection even of consummate politics becomes evident in religion . The so-called Christian state needs the Christian religion in order to complete itself as a state . The democratic state , the real state , does not need religion for its political completion . On the contrary , it can disregard religion because in it the human basis of religion is realized in a secular manner . The so-called Christian state , on the other hand , has a political attitude to religion and a religious attitude to politics . By degrading the forms of the state to mere sem
 blance , it equally degrades religion to mere semblance . In order to make this contradiction clearer , let us consider Bauer\u2019s projection of the Christian state , a projection based on his observation of the Christian-German state . \u201c Recently , \u201d says Bauer , \u201c in order to prove the impossibility or non-existence of a Christian state , reference has frequently been made to those sayings in the Gospel with which the [ present-day ] state not only does not comply , but cannot possibly comply , if it does not want to dissolve itself completely [ as a state ] . \u201d \u201c But the matter cannot be disposed of so easily . What do these Gospel sayings demand ? Supernatural renunciation of self , submission to the authority of revelation , a turning-away from the state , the abolition of secular conditions . Well , the Christian state demands and accomplishes all that . It has assimilated the spirit of the Gospel , and if it does not reproduce this spirit in the same terms as the Gos
 pel , that occurs only because it expresses this spirit in political forms , i. e. , in forms which , it is true , are taken from the political system in this world , but which in the religious rebirth that they have to undergo become degraded to a mere semblance . This is a turning-away from the state while making use of political forms for its realization . \u201d ( p. 55 ) Bauer then explains that the people of a Christian state is only a non-people , no longer having a will of its own , but whose true existence lies in the leader to whom it is subjected , although this leader by his origin and nature is alien to it \u2013 i. e. , given by God and imposed on the people without any co-operation on its part . Bauer declares that the laws of such a people are not its own creation , but are actual revelations , that its supreme chief needs privileged intermediaries with the people in the strict sense , with the masses , and that the masses themselves are divided into a multitude of particu
 lar groupings which are formed and determined by chance , which are differentiated by their interests , their particular passions and prejudices , and obtain permission as a privilege , to isolate themselves from one another , etc. ( p. 56 ) However , Bauer himself says : \u201c Politics , if it is to be nothing but religion , ought not to be politics , just as the cleaning of saucepans , if it is to be accepted as a religious matter , ought not to be regarded as a matter of domestic economy . \u201d ( p. 108 ) In the Christian-German state , however , religion is an \u201c economic matter \u201d just as \u201c economic matters \u201d belong to the sphere of religion . The domination of religion in the Christian-German state is the religion of domination . The separation of the \u201c spirit of the Gospel \u201d from the \u201c letter of the Gospel \u201d is an irreligious act . A state which makes the Gospel speak in the language of politics \u2013 that is , in another language than that of the Holy Ghost \u2013 comm
 its sacrilege , if not in human eyes , then in the eyes of its own religion . The state which acknowledges Christianity as its supreme criterion , and the Bible as its Charter , must be confronted with the words of Holy Scripture , for every word of Scripture is holy . This state , as well as the human rubbish on which it is based , is caught in a painful contradiction that is insoluble from the standpoint of religious consciousness when it is referred to those sayings of the Gospel with which it \u201c not only does not comply , but cannot possibly comply , if it does not want to dissolve itself completely as a state . \u201d And why does it not want to dissolve itself completely ? The state itself cannot give an answer either to itself or to others . In its own consciousness , the official Christian state is an imperative , the realization of which is unattainable , the state can assert the reality of its existence only by lying to itself , and therefore always remains in its own eyes a
 n object of doubt , an unreliable , problematic object . Criticism is , therefore , fully justified in forcing the state that relies on the Bible into a mental derangement in which it no longer knows whether it is an illusion or a reality , and in which the infamy of its secular aims , for which religion serves as a cloak , comes into insoluble conflict with the sincerity of its religious consciousness , for which religion appears as the aim of the world . This state can only save itself from its inner torment if it becomes the police agent of the Catholic Church . In relation to the church , which declares the secular power to be its servant , the state is powerless , the secular power which claims to be the rule of the religious spirit is powerless . It is , indeed , estrangement which matters in the so-called Christian state , but not man . The only man who counts , the king , is a being specifically different from other men , and is , moreover , a religious being , directly link
 ed with heaven , with God . The relationships which prevail here are still relationships dependent of faith . The religious spirit , therefore , is still not really secularized . But , furthermore , the religious spirit cannot be really secularized , for what is it in itself but the non-secular form of a stage in the development of the human mind ? The religious spirit can only be secularized insofar as the stage of development of the human mind of which it is the religious expression makes its appearance and becomes constituted in its secular form . This takes place in the democratic state . Not Christianity , but the human basis of Christianity is the basis of this state . Religion remains the ideal , non-secular consciousness of its members , because religion is the ideal form of the stage of human development achieved in this state . The members of the political state are religious owing to the dualism between individual life and species-life , between the life of civil society 
 and political life . They are religious because men treat the political life of the state , an area beyond their real individuality , as if it were their true life . They are religious insofar as religion here is the spirit of civil society , expressing the separation and remoteness of man from man . Political democracy is Christian since in it man , not merely one man but everyman , ranks as sovereign , as the highest being , but it is man in his uncivilized , unsocial form , man in his fortuitous existence , man just as he is , man as he has been corrupted by the whole organization of our society , who has lost himself , been alienated , and handed over to the rule of inhuman conditions and elements \u2013 in short , man who is not yet a real species-being . That which is a creation of fantasy , a dream , a postulate of Christianity , i. e. , the sovereignty of man \u2013 but man as an alien being different from the real man \u2013 becomes , in democracy , tangible reality , present existe
 nce , and secular principle . In the perfect democracy , the religious and theological consciousness itself is in its own eyes the more religious and the more theological because it is apparently without political significance , without worldly aims , the concern of a disposition that shuns the world , the expression of intellectual narrow-mindedness , the product of arbitrariness and fantasy , and because it is a life that is really of the other world . Christianity attains , here , the practical expression of its universal-religious significance in that the most diverse world outlooks are grouped alongside one another in the form of Christianity and still more because it does not require other people to profess Christianity , but only religion in general , any kind of religion ( cf . Beaumont\u2019s work quoted above ) . The religious consciousness revels in the wealth of religious contradictions and religious diversity . We have , thus , shown that political emancipation from religi
 on leaves religion in existence , although not a privileged religion . The contradiction in which the adherent of a particular religion finds himself involved in relation to his citizenship is only one aspect of the universal secular contradiction between the political state and civil society . The consummation of the Christian state is the state which acknowledges itself as a state and disregards the religion of its members . The emancipation of the state from religion is not the emancipation of the real man from religion . Therefore , we do not say to the Jews , as Bauer does : You cannot be emancipated politically without emancipating yourselves radically from Judaism . On the contrary , we tell them : Because you can be emancipated politically without renouncing Judaism completely and incontrovertibly , political emancipation itself is not human emancipation . If you Jews want to be emancipated politically , without emancipating yourselves humanly , the half-hearted approach and
  contradiction is not in you alone , it is inherent in the nature and category of political emancipation . If you find yourself within the confines of this category , you share in a general confinement . Just as the state evangelizes when , although it is a state , it adopts a Christian attitude towards the Jews , so the Jew acts politically when , although a Jew , he demands civic rights . But , if a man , although a Jew , can be emancipated politically and receive civic rights , can he lay claim to the so-called rights of man and receive them ? Bauer denies it . \u201c The question is whether the Jew as such , that is , the Jew who himself admits that he is compelled by his true nature to live permanently in separation from other men , is capable of receiving the universal rights of man and of conceding them to others . \u201d \u201c For the Christian world , the idea of the rights of man was only discovered in the last century . It is not innate in men ; on the contrary , it is gained onl
 y in a struggle against the historical traditions in which hitherto man was brought up . Thus the rights of man are not a gift of nature , not a legacy from past history , but the reward of the struggle against the accident of birth and against the privileges which up to now have been handed down by history from generation to generation . These rights are the result of culture , and only one who has earned and deserved them can possess them . \u201d \u201c Can the Jew really take possession of them ? As long as he is a Jew , the restricted nature which makes him a Jew is bound to triumph over the human nature which should link him as a man with other men , and will separate him from non-Jews . He declares by this separation that the particular nature which makes him a Jew is his true , highest nature , before which human nature has to give way . \u201d \u201c Similarly , the Christian as a Christian cannot grant the rights of man . \u201d ( p. 19-20 ) According to Bauer , man has to sacrifice the 
 \u201c privilege of faith \u201d to be able to receive the universal rights of man . Let us examine , for a moment , the so-called rights of man \u2013 to be precise , the rights of man in their authentic form , in the form which they have among those who discovered them , the North Americans and the French . These rights of man are , in part , political rights , rights which can only be exercised in community with others . Their content is participation in the community , and specifically in the political community , in the life of the state . They come within the category of political freedom , the category of civic rights , which , as we have seen , in no way presuppose the incontrovertible and positive abolition of religion \u2013 nor , therefore , of Judaism . There remains to be examined the other part of the rights of man \u2013 the droits d\u2019homme , insofar as these differ from the droits d\u2019citoyen . Included among them is freedom of conscience , the right to practice any religion one c
 hooses . The privilege of faith is expressly recognized either as a right of man or as the consequence of a right of man , that of liberty . D�claration des droits de l\u2019droits et du citoyen , 1791 , Article 10 : \u201c No one is to be subjected to annoyance because of his opinions , even religious opinions . \u201d \u201c The freedom of every man to practice the religion of which he is an adherent . \u201d Declaration of the Rights of Man , etc. , 1793 , includes among the rights of man , Article 7 : \u201c The free exercise of religion . \u201d Indeed , in regard to man\u2019s right to express his thoughts and opinions , to hold meetings , and to exercise his religion , it is even stated : \u201c The necessity of proclaiming these rights presupposes either the existence or the recent memory of despotism . \u201d Compare the Constitution of 1795 , Section XIV , Article 354. Constitution of Pennsylvania , Article 9 , � 3 : \u201c All men have received from nature the imprescriptible right to worship the Almig
 hty according to the dictates of their conscience , and no one can be legally compelled to follow , establish , or support against his will any religion or religious ministry . No human authority can , in any circumstances , intervene in a matter of conscience or control the forces of the soul . \u201d Constitution of New Hampshire , Article 5 and 6 : \u201c Among these natural rights some are by nature inalienable since nothing can replace them . The rights of conscience are among them . \u201d ( Beaumont , op . cit . , pp . 213,214 ) Incompatibility between religion and the rights of man is to such a degree absent from the concept of the rights of man that , on the contrary , a man\u2019s right to be religious , in any way he chooses , to practise his own particular religion , is expressly included among the rights of man . The privilege of faith is a universal right of man . The droits de l\u2019homme , the rights of man , are , as such , distinct from the droits du citoyen , the rights of the 
 citizen . Who is homme as distinct from citoyen ? None other than the member of civil society . Why is the member of civil society called \u201c man , \u201d simply man ; why are his rights called the rights of man ? How is this fact to be explained ? From the relationship between the political state and civil society , from the nature of political emancipation . Above all , we note the fact that the so-called rights of man , the droits de l\u2019homme as distinct from the droits du citoyen , are nothing but the rights of a member of civil society \u2013 i. e. , the rights of egoistic man , of man separated from other men and from the community . Let us hear what the most radical Constitution , the Constitution of 1793 , has to say : Declaration of the Rights of Man and of the Citizen . Article 2. \u201c These rights , etc. , ( the natural and imprescriptible rights ) are : equality , liberty , security , property . \u201d What constitutes liberty ? Article 6. \u201c Liberty is the power which man has t
 o do everything that does not harm the rights of others , \u201d or , according to the Declaration of the Rights of Man of 1791 : \u201c Liberty consists in being able to do everything which does not harm others . \u201d Liberty , therefore , is the right to do everything that harms no one else . The limits within which anyone can act without harming someone else are defined by law , just as the boundary between two fields is determined by a boundary post . It is a question of the liberty of man as an isolated monad , withdrawn into himself . Why is the Jew , according to Bauer , incapable of acquiring the rights of man ? \u201c As long as he is a Jew , the restricted nature which makes him a Jew is bound to triumph over the human nature which should link him as a man with other men , and will separate him from non-Jews . \u201d But , the right of man to liberty is based not on the association of man with man , but on the separation of man from man . It is the right of this separation , the right 
 of the restricted individual , withdrawn into himself . The practical application of man\u2019s right to liberty is man\u2019s right to private property . What constitutes man\u2019s right to private property ? Article 16. ( Constitution of 1793 ) : \u201c The right of property is that which every citizen has of enjoying and of disposing at his discretion of his goods and income , of the fruits of his labor and industry . \u201d The right of man to private property is , therefore , the right to enjoy one\u2019s property and to dispose of it at one\u2019s discretion ( � son gr� ) , without regard to other men , independently of society , the right of self-interest . This individual liberty and its application form the basis of civil society . It makes every man see in other men not the realization of his own freedom , but the barrier to it . But , above all , it proclaims the right of man \u201c of enjoying and of disposing at his discretion of his goods and income , of the fruits of his labor and industr
 y . \u201d There remain the other rights of man : �galit� and s�ret� . Equality , used here in its non-political sense , is nothing but the equality of the libert� described above \u2013 namely : each man is to the same extent regarded as such a self-sufficient monad . The Constitution of 1795 defines the concept of this equality , in accordance with this significance , as follows : Article 3 ( Constitution of 1795 ) : \u201c Equality consists in the law being the same for all , whether it protects or punishes . \u201d And security ? Article 8 ( Constitution of 1793 ) : \u201c Security consists in the protection afforded by society to each of its members for the preservation of his person , his rights , and his property . \u201d Security is the highest social concept of civil society , the concept of police , expressing the fact that the whole of society exists only in order to guarantee to each of its members the preservation of his person , his rights , and his property . It is in this sense t
 hat Hegel calls civil society \u201c the state of need and reason . \u201d The concept of security does not raise civil society above its egoism . On the contrary , security is the insurance of egoism . None of the so-called rights of man , therefore , go beyond egoistic man , beyond man as a member of civil society \u2013 that is , an individual withdrawn into himself , into the confines of his private interests and private caprice , and separated from the community . In the rights of man , he is far from being conceived as a species-being ; on the contrary , species-like itself , society , appears as a framework external to the individuals , as a restriction of their original independence . The sole bond holding them together is natural necessity , need and private interest , the preservation of their property and their egoistic selves . It is puzzling enough that a people which is just beginning to liberate itself , to tear down all the barriers between its various sections , and to estab
 lish a political community , that such a people solemnly proclaims ( Declaration of 1791 ) the rights of egoistic man separated from his fellow men and from the community , and that indeed it repeats this proclamation at a moment when only the most heroic devotion can save the nation , and is therefore imperatively called for , at a moment when the sacrifice of all the interest of civil society must be the order of the day , and egoism must be punished as a crime . ( Declaration of the Rights of Man , etc. , of 1793 ) This fact becomes still more puzzling when we see that the political emancipators go so far as to reduce citizenship , and the political community , to a mere means for maintaining these so-called rights of man , that , therefore , the citoyen is declared to be the servant of egotistic homme , that the sphere in which man acts as a communal being is degraded to a level below the sphere in which he acts as a partial being , and that , finally , it is not man as citoyen 
 , but man as private individual [ bourgeois ] who is considered to be the essential and true man . \u201c The aim of all political association is the preservation of the natural and imprescriptible rights of man . \u201d ( Declaration of the Rights , etc. , of 1791 , Article 2 ) \u201c Government is instituted in order to guarantee man the enjoyment of his natural and imprescriptible rights . \u201d ( Declaration , etc. , of 1793 , Article 1 ) Hence , even in moments when its enthusiasm still has the freshness of youth and is intensified to an extreme degree by the force of circumstances , political life declares itself to be a mere means , whose purpose is the life of civil society . It is true that its revolutionary practice is in flagrant contradiction with its theory . Whereas , for example , security is declared one of the rights of man , violation of the privacy of correspondence is openly declared to be the order of the day . Whereas \u201c unlimited freedom of the press \u201d ( Constitution 
 of 1793 , Article 122 ) is guaranteed as a consequence of the right of man to individual liberty , freedom of the press is totally destroyed , because \u201c freedom of the press should not be permitted when it endangers public liberty . \u201d ( \u201c Robespierre jeune , \u201d Historie parlementaire de la R�volution fran�aise by Buchez and Roux , vol.28 , p. 159 ) That is to say , therefore : The right of man to liberty ceases to be a right as soon as it comes into conflict with political life , whereas in theory political life is only the guarantee of human rights , the rights of the individual , and therefore must be abandoned as soon as it comes into contradiction with its aim , with these rights of man . But , practice is merely the exception , theory is the rule . But even if one were to regard revolutionary practice as the correct presentation of the relationship , there would still remain the puzzle of why the relationship is turned upside-down in the minds of the political emancipa
 tors and the aim appears as the means , while the means appears as the aim . This optical illusion of their consciousness would still remain a puzzle , although now a psychological , a theoretical puzzle . The puzzle is easily solved . Political emancipation is , at the same time , the dissolution of the old society on which the state alienated from the people , the sovereign power , is based . What was the character of the old society ? It can be described in one word \u2013 feudalism . The character of the old civil society was directly political \u2013 that is to say , the elements of civil life , for example , property , or the family , or the mode of labor , were raised to the level of elements of political life in the form of seigniory , estates , and corporations . In this form , they determined the relation of the individual to the state as a whole \u2013 i. e. , his political relation , that is , his relation of separation and exclusion from the other components of society . For tha
 t organization of national life did not raise property or labor to the level of social elements ; on the contrary , it completed their separation from the state as a whole and constituted them as discrete societies within society . Thus , the vital functions and conditions of life of civil society remained , nevertheless , political , although political in the feudal sense \u2013 that is to say , they secluded the individual from the state as a whole and they converted the particular relation of his corporation to the state as a whole into his general relation to the life of the nation , just as they converted his particular civil activity and situation into his general activity and situation . As a result of this organization , the unity of the state , and also the consciousness , will , and activity of this unity , the general power of the state , are likewise bound to appear as the particular affair of a ruler and of his servants , isolated from the people . The political revolution
  which overthrew this sovereign power and raised state affairs to become affairs of the people , which constituted the political state as a matter of general concern , that is , as a real state , necessarily smashed all estates , corporations , guilds , and privileges , since they were all manifestations of the separation of the people from the community . The political revolution thereby abolished the political character of civil society . It broke up civil society into its simple component parts ; on the one hand , the individuals ; on the other hand , the material and spiritual elements constituting the content of the life and social position of these individuals . It set free the political spirit , which had been , as it were , split up , partitioned , and dispersed in the various blind alleys of feudal society . It gathered the dispersed parts of the political spirit , freed it from its intermixture with civil life , and established it as the sphere of the community , the gener
 al concern of the nation , ideally independent of those particular elements of civil life . A person\u2019s distinct activity and distinct situation in life were reduced to a merely individual significance . They no longer constituted the general relation of the individual to the state as a whole . Public affairs as such , on the other hand , became the general affair of each individual , and the political function became the individual\u2019s general function . But , the completion of the idealism of the state was at the same time the completion of the materialism of civil society . Throwing off the political yoke meant at the same time throwing off the bonds which restrained the egoistic spirit of civil society . Political emancipation was , at the same time , the emancipation of civil society from politics , from having even the semblance of a universal content . Feudal society was resolved into its basic element \u2013 man , but man as he really formed its basis \u2013 egoistic man . This m
 an , the member of civil society , is thus the basis , the precondition , of the political state . He is recognized as such by this state in the rights of man . The liberty of egoistic man and the recognition of this liberty , however , is rather the recognition of the unrestrained movement of the spiritual and material elements which form the content of his life . Hence , man was not freed from religion , he received religious freedom . He was not freed from property , he received freedom to own property . He was not freed from the egoism of business , he received freedom to engage in business . The establishment of the political state and the dissolution of civil society into independent individuals \u2013 whose relation with one another on law , just as the relations of men in the system of estates and guilds depended on privilege \u2013 is accomplished by one and the same act . Man as a member of civil society , unpolitical man , inevitably appears , however , as the natural man . The
  \u201c rights of man \u201d appears as \u201c natural rights , \u201d because conscious activity is concentrated on the political act . Egoistic man is the passive result of the dissolved society , a result that is simply found in existence , an object of immediate certainty , therefore a natural object . The political revolution resolves civil life into its component parts , without revolutionizing these components themselves or subjecting them to criticism . It regards civil society , the world of needs , labor , private interests , civil law , as the basis of its existence , as a precondition not requiring further substantiation and therefore as its natural basis . Finally , man as a member of civil society is held to be man in his sensuous , individual , immediate existence , whereas political man is only abstract , artificial man , man as an allegorical , juridical person . The real man is recognized only in the shape of the egoistic individual , the true man is recognized only in the sha
 pe of the abstract citizen . Therefore , Rousseau correctly described the abstract idea of political man as follows : \u201c Whoever dares undertake to establish a people\u2019s institutions must feel himself capable of changing , as it were , human nature , of transforming each individual , who by himself is a complete and solitary whole , into a part of a larger whole , from which , in a sense , the individual receives his life and his being , of substituting a limited and mental existence for the physical and independent existence . He has to take from man his own powers , and give him in exchange alien powers which he cannot employ without the help of other men . \u201d All emancipation is a reduction of the human world and relationships to man himself . Political emancipation is the reduction of man , on the one hand , to a member of civil society , to an egoistic , independent individual , and , on the other hand , to a citizen , a juridical person . Only when the real , individual man
  re-absorbs in himself the abstract citizen , and as an individual human being has become a species-being in his everyday life , in his particular work , and in his particular situation , only when man has recognized and organized his \u201c own powers \u201d as social powers , and , consequently , no longer separates social power from himself in the shape of political power , only then will human emancipation have been accomplished . II Bruno Bauer , \u201c The Capacity of Present-day Jews and Christians to Become Free , \u201d Einundzwanzig Bogen aus der Schweiz , pp . 56-71 It is in this form that Bauer deals with the relation between the Jewish and the Christian religions , and also with their relation to criticism . Their relation to criticism is their relation \u201c to the capacity to become free . \u201d The result arrived at is : \u201c The Christian has to surmount only one stage , namely , that of his religion , in order to give up religion altogether , \u201d and therefore become free . \u201c The
  Jew , on the other hand , has to break not only with his Jewish nature , but also with the development towards perfecting his religion , a development which has remained alien to him . \u201d ( p. 71 ) Thus , Bauer here transforms the question of Jewish emancipation into a purely religious question . The theological problem as to whether the Jew or the Christian has the better prospect of salvation is repeated here in the enlightened form : which of them is more capable of emancipation . No longer is the question asked : Is it Judaism or Christianity that makes a man free ? On the contrary , the question is now : Which makes man freer , the negation of Judaism or the negation of Christianity ? \u201c If the Jews want to become free , they should profess belief not in Christianity , but in the dissolution of Christianity , in the dissolution of religion in general , that is to say , in enlightenment , criticism , and its consequences , free humanity . \u201d ( p. 70 ) For the Jew , it is sti
 ll a matter of a profession of faith , but no longer a profession of belief in Christianity , but of belief in Christianity in dissolution . Bauer demands of the Jews that they should break with the essence of the Christian religion , a demand which , as he says himself , does not arise out of the development of Judaism . Since Bauer , at the end of his work on the Jewish question , had conceived Judaism only as crude religious criticism of Christianity , and therefore saw in it \u201c merely \u201d a religious significance , it could be foreseen that the emancipation of the Jews , too , would be transformed into a philosophical-theological act . Bauer considers that the ideal , abstract nature of the Jew , his religion , is his entire nature . Hence , he rightly concludes : \u201c The Jew contributes nothing to mankind if he himself disregards his narrow law , \u201d if he invalidates his entire Judaism . ( p. 65 ) Accordingly , the relation between Jews and Christians becomes the following : 
 the sole interest of the Christian in the emancipation of the Jew is a general human interest , a theoretical interest . Judaism is a fact that offends the religious eye of the Christian . As soon as his eye ceases to be religious , this fact ceases to be offensive . The emancipation of the Jew is , in itself , not a task for the Christian . The Jew , on the other hand , in order to emancipate himself , has to carry out not only his own work , but also that of the Christian \u2013 i. e. , the Critique of the Evangelical History of the Synoptics and the Life of Jesus , etc. \u201c It is up to them to deal with it : they themselves will decide their fate ; but history is not to be trifled with . \u201d ( p. 71 ) We are trying to break with the theological formulation of the question . For us , the question of the Jew\u2019s capacity for emancipation becomes the question : What particular social element has to be overcome in order to abolish Judaism ? For the present-day Jew\u2019s capacity for emanc
 ipation is the relation of Judaism to the emancipation of the modern world . This relation necessarily results from the special position of Judaism in the contemporary enslaved world . Let us consider the actual , worldly Jew \u2013 not the Sabbath Jew , as Bauer does , but the everyday Jew . Let us not look for the secret of the Jew in his religion , but let us look for the secret of his religion in the real Jew . What is the secular basis of Judaism ? Practical need , self-interest . What is the worldly religion of the Jew ? Huckstering . What is his worldly God ? Money . Very well then ! Emancipation from huckstering and money , consequently from practical , real Judaism , would be the self-emancipation of our time . An organization of society which would abolish the preconditions for huckstering , and therefore the possibility of huckstering , would make the Jew impossible . His religious consciousness would be dissipated like a thin haze in the real , vital air of society . On the
  other hand , if the Jew recognizes that this practical nature of his is futile and works to abolish it , he extricates himself from his previous development and works for human emancipation as such and turns against the supreme practical expression of human self-estrangement . We recognize in Judaism , therefore , a general anti-social element of the present time , an element which through historical development \u2013 to which in this harmful respect the Jews have zealously contributed \u2013 has been brought to its present high level , at which it must necessarily begin to disintegrate . In the final analysis , the emancipation of the Jews is the emancipation of mankind from Judaism . The Jew has already emancipated himself in a Jewish way . \u201c The Jew , who in Vienna , for example , is only tolerated , determines the fate of the whole Empire by his financial power . The Jew , who may have no rights in the smallest German state , decides the fate of Europe . While corporations and gui
 lds refuse to admit Jews , or have not yet adopted a favorable attitude towards them , the audacity of industry mocks at the obstinacy of the material institutions . \u201d ( Bruno Bauer , The Jewish Question , p. 114 ) This is no isolated fact . The Jew has emancipated himself in a Jewish manner , not only because he has acquired financial power , but also because , through him and also apart from him , money has become a world power and the practical Jewish spirit has become the practical spirit of the Christian nations . The Jews have emancipated themselves insofar as the Christians have become Jews . Captain Hamilton , for example , reports : \u201c The devout and politically free inhabitant of New England is a kind of Laoco�n who makes not the least effort to escape from the serpents which are crushing him . Mammon is his idol which he adores not only with his lips but with the whole force of his body and mind . In his view the world is no more than a Stock Exchange , and he is conv
 inced that he has no other destiny here below than to become richer than his neighbor . Trade has seized upon all his thoughts , and he has no other recreation than to exchange objects . When he travels he carries , so to speak , his goods and his counter on his back and talks only of interest and profit . If he loses sight of his own business for an instant it is only in order to pry into the business of his competitors . \u201d Indeed , in North America , the practical domination of Judaism over the Christian world has achieved as its unambiguous and normal expression that the preaching of the Gospel itself and the Christian ministry have become articles of trade , and the bankrupt trader deals in the Gospel just as the Gospel preacher who has become rich goes in for business deals . \u201c The man who you see at the head of a respectable congregation began as a trader ; his business having failed , he became a minister . The other began as a priest but as soon as he had some money at h
 is disposal he left the pulpit to become a trader . In the eyes of very many people , the religious ministry is a veritable business career . \u201d ( Beaumont , op . cit . , pp . 185,186 ) According to Bauer , it is \u201c a fictitious state of affairs when in theory the Jew is deprived of political rights , whereas in practice he has immense power and exerts his political influence en gros , although it is curtailed en d�tail . \u201d ( Die Judenfrage , p. 114 ) The contradiction that exists between the practical political power of the Jew and his political rights is the contradiction between politics and the power of money in general . Although theoretically the former is superior to the latter , in actual fact politics has become the serf of financial power . Judaism has held its own alongside Christianity , not only as religious criticism of Christianity , not only as the embodiment of doubt in the religious derivation of Christianity , but equally because the practical Jewish spirit ,
  Judaism , has maintained itself and even attained its highest development in Christian society . The Jew , who exists as a distinct member of civil society , is only a particular manifestation of the Judaism of civil society . Judaism continues to exist not in spite of history , but owing to history . The Jew is perpetually created by civil society from its own entrails . What , in itself , was the basis of the Jewish religion ? Practical need , egoism . The monotheism of the Jew , therefore , is in reality the polytheism of the many needs , a polytheism which makes even the lavatory an object of divine law . Practical need , egoism , is the principle of civil society , and as such appears in pure form as soon as civil society has fully given birth to the political state . The god of practical need and self-interest is money . Money is the jealous god of Israel , in face of which no other god may exist . Money degrades all the gods of man \u2013 and turns them into commodities . Money
  is the universal self-established value of all things . It has , therefore , robbed the whole world \u2013 both the world of men and nature \u2013 of its specific value . Money is the estranged essence of man\u2019s work and man\u2019s existence , and this alien essence dominates him , and he worships it . The god of the Jews has become secularized and has become the god of the world . The bill of exchange is the real god of the Jew . His god is only an illusory bill of exchange . The view of nature attained under the domination of private property and money is a real contempt for , and practical debasement of , nature ; in the Jewish religion , nature exists , it is true , but it exists only in imagination . It is in this sense that [ in a 1524 pamphlet ] Thomas M�nzer declares it intolerable \u201c that all creatures have been turned into property , the fishes in the water , the birds in the air , the plants on the earth ; the creatures , too , must become free . \u201d Contempt for theory , art 
 , history , and for man as an end in himself , which is contained in an abstract form in the Jewish religion , is the real , conscious standpoint , the virtue of the man of money . The species-relation itself , the relation between man and woman , etc. , becomes an object of trade ! The woman is bought and sold . The chimerical nationality of the Jew is the nationality of the merchant , of the man of money in general . The groundless law of the Jew is only a religious caricature of groundless morality and right in general , of the purely formal rites with which the world of self-interest surrounds itself . Here , too , man\u2019s supreme relation is the legal one , his relation to laws that are valid for him not because they are laws of his own will and nature , but because they are the dominant laws and because departure from them is avenged . Jewish Jesuitism , the same practical Jesuitism which Bauer discovers in the Talmud , is the relation of the world of self-interest to the laws
  governing that world , the chief art of which consists in the cunning circumvention of these laws . Indeed , the movement of this world within its framework of laws is bound to be a continual suspension of law . Judaism could not develop further as a religion , could not develop further theoretically , because the world outlook of practical need is essentially limited and is completed in a few strokes . By its very nature , the religion of practical need could find its consummation not in theory , but only in practice , precisely because its truth is practice . Judaism could not create a new world ; it could only draw the new creations and conditions of the world into the sphere of its activity , because practical need , the rationale of which is self-interest , is passive and does not expand at will , but finds itself enlarged as a result of the continuous development of social conditions . Judaism reaches its highest point with the perfection of civil society , but it is only in 
 the Christian world that civil society attains perfection . Only under the dominance of Christianity , which makes all national , natural , moral , and theoretical conditions extrinsic to man , could civil society separate itself completely from the life of the state , sever all the species-ties of man , put egoism and selfish need in the place of these species-ties , and dissolve the human world into a world of atomistic individuals who are inimically opposed to one another . Christianity sprang from Judaism . It has merged again in Judaism . From the outset , the Christian was the theorizing Jew , the Jew is , therefore , the practical Christian , and the practical Christian has become a Jew again . Christianity had only in semblance overcome real Judaism . It was too noble-minded , too spiritualistic to eliminate the crudity of practical need in any other way than by elevation to the skies . Christianity is the sublime thought of Judaism , Judaism is the common practical applicat
 ion of Christianity , but this application could only become general after Christianity as a developed religion had completed theoretically the estrangement of man from himself and from nature . Only then could Judaism achieve universal dominance and make alienated man and alienated nature into alienable , vendible objects subjected to the slavery of egoistic need and to trading . Selling [ Ver�u�erung ] is the practical aspect of alienation [ Ent�u�erung ] . Just as man , as long as he is in the grip of religion , is able to objectify his essential nature only by turning it into something alien , something fantastic , so under the domination of egoistic need he can be active practically , and produce objects in practice , only by putting his products , and his activity , under the domination of an alien being , and bestowing the significance of an alien entity \u2013 money \u2013 on them . In its perfected practice , Christian egoism of heavenly bliss is necessarily transformed into 
 the corporal egoism of the Jew , heavenly need is turned into world need , subjectivism into self-interest . We explain the tenacity of the Jew not by his religion , but , on the contrary , by the human basis of his religion \u2013 practical need , egoism . Since in civil society the real nature of the Jew has been universally realized and secularized , civil society could not convince the Jew of the unreality of his religious nature , which is indeed only the ideal aspect of practical need . Consequently , not only in the Pentateuch and the Talmud , but in present-day society we find the nature of the modern Jew , and not as an abstract nature but as one that is in the highest degree empirical , not merely as a narrowness of the Jew , but as the Jewish narrowness of society . Once society has succeeded in abolishing the empirical essence of Judaism \u2013 huckstering and its preconditions \u2013 the Jew will have become impossible , because his consciousness no longer has an object , becaus
 e the subjective basis of Judaism , practical need , has been humanized , and because the conflict between man\u2019s individual-sensuous existence and his species-existence has been abolished . The social emancipation of the Jew is the emancipation of society from Judaism . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/News/494news1\" url=\"http:__observer.guardian.co.uk_international_story_0,,631931,00.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/News/495news2\" url=\"http:__news.bbc.co.uk_2_hi_middle_east_5136988.stm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/News/496news3\" url=\"http:__news.bbc.co.uk_2_hi_asia-pacific_2181151.stm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/News/497news4\" url=\"http:__news.bbc.co.uk_2_hi_business_2235352.stm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/News/498news5\" url=\"http:__money.guardian.co.uk_news__0,,,00.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/News/499news6\" url=\"http:__football.guardian.co.uk_Match_Report_0,,1536875,00.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/News/500news7\" url=\"http:__news.bbc.co.uk_2_hi_asia-pacific_5133220.stm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/News/501news8\" url=\"http:__news.bbc.co.uk_2_hi_europe_4263426.stm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/News/502news9\" url=\"http:__news.bbc.co.uk_2_hi_business_2769931.stm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/News/503newsa\" url=\"http:__news.bbc.co.uk_2_hi_uk_news_education_2191422.stm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/News/504newsb\" url=\"http:__news.bbc.co.uk_2_hi_africa_2909769.stm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/News/505newsc\" url=\"http:__news.bbc.co.uk_2_hi_uk_news_2509561.stm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/79NewsEnnajiM_Maghreb_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/News/79NewsEnnajiM_Maghreb_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/79NewsEnnajiM_Maghreb_EN.txt.txt
new file mode 100644
index 0000000..5acd175
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/79NewsEnnajiM_Maghreb_EN.txt.txt
@@ -0,0 +1,2 @@
+
+The Maghreb\u2019s Modern Islamists Moha Ennaji 2012-02-16 FEZ \u2013 Just over a year ago , the Arab Spring sparked dramatic change throughout the Arab world . Popular movements have brought a range of avowedly Islamist political parties to power , replacing the largely secular former regimes . What that will mean for these countries , and for the region , is one of today\u2019s central geopolitical questions . In North Africa , two Islamist parties have come fully to power via democratic elections : al-Nahda ( Renaissance ) in Tunisia , where the Arab Spring began , and the Justice and Development Party ( PJD ) in Morocco , both of which now lead new coalition governments . Whereas a popular revolution produced regime change in Tunisia , Morocco underwent a peaceful transformation that left the monarchy in place . Last July , Moroccans voted overwhelmingly to approve a new constitution that shifts executive power from the king to the prime minister , who will now be fully responsible for t
 he cabinet , the civil service , and the implementation of government policies . The king retains some prerogatives , such as the authority to select the prime minister ( from the parliament\u2019s majority party ) and the head of the army . Furthermore , like heads of state in other parliamentary systems , he has the right to appoint the government ministers and ambassadors , dissolve the parliament , and dismiss the cabinet . While the PJD\u2019s success was based on institution-building within a constitutional monarchy , al-Nahda has revolutionized the Tunisian system . But both parties won after running on a moderate platform of constitutionalism , separation of powers , civil liberties , and women\u2019s rights . This new political reality in the Maghreb will bring Europe \u2013 particularly France , the region\u2019s old colonial master \u2013 face-to-face with Islamist governments determined to promote a new type of relationship . But these governments have much work to do at home first . Curr
 ently , the Maghreb countries suffer from soaring unemployment , poverty , and high prices for basic commodities . In response , both al-Nahda and the PJD are emphasizing job creation , free trade , foreign investment , and a crackdown on the corruption that has plagued their countries \u2019 economies . These governments \u2019 first major test of economic stewardship will be how they treat the tourism industry . Although Western tourism is a critically important source of employment and foreign currency in both countries , some Muslims have criticized the industry for promoting alcohol and other relaxed social conventions that threaten Islamic values . So far , both al-Nahda and the PJD have taken a pragmatic stance . They recognize that , while their supporters may be devout Muslims , they also need to earn a living ; empty hotels and beaches would be economically disastrous . Thus , tourism professionals in both countries have received strong government assurances that business will c
 ontinue as usual . Some European analysts predict that , over the longer term , greater stability will follow the political changes in the Maghreb , with perhaps more than a million unemployed Moroccan and Tunisian immigrants returning home if their countries \u2019 economies improve . That remains very much an open question . Islamist parties will now have enormous influence on economic policy , after decades of official separation of mosque and state . Islamic banking , for example , may soon be introduced , though some local and foreign investors argue that sharia regulations could drive away much-needed foreign investment . There are also concerns about inexperienced Islamist officials \u2019 ability to run finance ministries . But the region\u2019s Islamist parties appear to be conscious of these risks , and determined to mitigate them . They know that they need economic growth to curb unemployment and pay for social services , so they are working to bolster the private sector . In many
  cases , they are even advocating the kind of free-market policies that their secular predecessors favored . Those policies should include trade liberalization . Until now , less than 2 % of the Maghreb countries \u2019 foreign trade has remained within the region . If the region\u2019s new leaders can integrate their economies , a market of more than 75 million consumers would attract more foreign investment and trade with the rest of the world . Before an economically unified Maghreb can be realized , however , inter-state conflicts such as the Algerian-Moroccan dispute over the \u201c Western \u201d Sahara must be resolved . Otherwise , it will be difficult even to conceive a common future \u2013 without which the economic grievances that fueled the Maghreb\u2019s revolutions are likely to continue . Much like the Muslim Brotherhood in Egypt , al-Nahda and the PJD will have to marginalize Islamic extremists in their movements , such as the Salafis , and adopt a pragmatic approach . To succeed econ
 omically , they will need Western support \u2013 and the West will not be eager to fund radical Islamist governments . As they negotiate the realities of modern economic life , the Maghreb\u2019s Islamist ruling parties are likely to lose some supporters . But , unless they are willing to break with the past , they will not succeed in the present . 
\ No newline at end of file


[10/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/157ted_isabel_allende_tells_tales_of_passion.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/157ted_isabel_allende_tells_tales_of_passion.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/157ted_isabel_allende_tells_tales_of_passion.txt
new file mode 100644
index 0000000..a9a8f8a
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/157ted_isabel_allende_tells_tales_of_passion.txt
@@ -0,0 +1,2 @@
+
+Thank you so much . It 's really scary to be here among the smartest of the smart . I 'm here to tell you a few tales of passion . There 's a Jewish saying that I love . What is truer than truth ? Answer : The story . I 'm a storyteller . I want to convey something that is truer than truth about our common humanity . All stories interest me , and some haunt me until I end up writing them . Certain themes keep coming up : justice , loyalty , violence , death , political and social issues , freedom . I 'm aware of the mystery around us , so I write about coincidences , premonitions , emotions , dreams , the power of nature , magic . In the last 20 years I have published a few books , but I have lived in anonymity until February of 2006 , when I carried the Olympic flag in the Winter Olympics in Italy . That made me a celebrity . Now people recognize me in Macy 's , and my grandchildren think that I 'm cool . ( Laughter ) Allow me to tell you about my four minutes of fame . One of the 
 organizers of the Olympic ceremony , of the opening ceremony , called me and said that I had been selected to be one of the flag-bearers . I replied that surely this was a case of mistaken identity because I 'm as far as you can get from being an athlete . Actually , I was n't even sure that I could go around the stadium without a walker . ( Laughter ) I was told that this was no laughing matter . This would be the first time that only women would carry the Olympic flag . Five women , representing five continents , and three Olympic gold medal winners . My first question was , naturally , what was I going to wear ? ( Laughter ) A uniform , she said , and asked for my measurements . My measurements . I had a vision of myself in a fluffy anorak , looking like the Michelin Man . ( Laughter ) By the middle of February , I found myself in Turin , where enthusiastic crowds cheered when any of the 80 Olympic teams was in the street . Those athletes had sacrificed everything to compete in t
 he games . They all deserved to win , but there 's the element of luck . A speck of snow , an inch of ice , the force of the wind , can determine the result of a race or a game . However , what matters most -- more than training or luck -- is the heart . Only a fearless and determined heart will get the gold medal . It is all about passion . The streets of Turin were covered with red posters announcing the slogan of the Olympics . Passion lives here . Is n't it always true ? Heart is what drives us and determines our fate . That is what I need for my characters in my books : a passionate heart . I need mavericks , dissidents , adventurers , outsiders and rebels , who ask questions , bend the rules and take risks . People like all of you in this room . Nice people with common sense do not make interesting characters . ( Laughter ) They only make good former spouses . ( Laughter ) ( Applause ) In the green room of the stadium , I met the other flag bearers : three athletes , and the a
 ctresses Susan Sarandon and Sophia Loren . Also , two women with passionate hearts . Wangari Maathai , the Nobel prizewinner from Kenya who has planted 30 million trees . And by doing so , she has changed the soil , the weather , in some places in Africa , and of course the economic conditions in many villages . And Somaly Mam , a Cambodian activist who fights passionately against child prostitution . When she was 14 years old , her grandfather sold her to a brothel . She told us of little girls raped by men who believe that having sex with a very young virgin will cure them from AIDS . And of brothels where children are forced to receive five , 15 clients per day , and if they rebel , they are tortured with electricity . In the green room I received my uniform . It was not the kind of outfit that I normally wear , but it was far from the Michelin Man suit that I had anticipated . Not bad , really . I looked like a refrigerator . ( Laughter ) But so did most of the flag-bearers , ex
 cept Sophia Loren , the universal symbol of beauty and passion . Sophia is over 70 and she looks great . She 's sexy , slim and tall , with a deep tan . Now , how can you have a deep tan and have no wrinkles ? I do n't know . When asked in a TV interview , " How could she look so good ? " She replied , " Posture . My back is always straight , and I do n't make old people 's noises . " ( Laughter ) So , there you have some free advice from one of the most beautiful women on earth . No grunting , no coughing , no wheezing , no talking to yourselves , no farting . ( Laughter ) Well , she did n't say that exactly . ( Laughter ) At some point around midnight , we were summoned to the wings of the stadium , and the loudspeakers announced the Olympic flag , and the music started -- by the way , the same music that starts here , the Aida March . Sophia Loren was right in front of me -- she 's a foot taller than I am , not counting the poofy hair . ( Laughter ) She walked elegantly , like a 
 giraffe on the African savannah , holding the flag on her shoulder . I jogged behind -- ( Laughter ) -- on my tiptoes , holding the flag on my extended arm , so that my head was actually under the damn flag . ( Laughter ) All the cameras were , of course , on Sophia . That was fortunate for me , because in most press photos I appear too , although often between Sophia 's legs . ( Laughter ) A place where most men would love to be . ( Laughter ) ( Applause ) The best four minutes of my entire life were those in the Olympic stadium . My husband is offended when I say this -- although I have explained to him that what we do in private usually takes less than four minutes -- ( Laughter ) -- so he should n't take it personally . I have all the press clippings of those four magnificent minutes , because I do n't want to forget them when old age destroys my brain cells . I want to carry in my heart forever the key word of the Olympics -- passion . So here 's a tale of passion . The year is
  1998 , the place is a prison camp for Tutsi refugees in Congo . By the way , 80 percent of all refugees and displaced people in the world are women and girls . We can call this place in Congo a death camp , because those who are not killed will die of disease or starvation . The protagonists of this story are a young woman , Rose Mapendo , and her children . She 's pregnant and a widow . Soldiers have forced her to watch as her husband was tortured and killed . Somehow she manages to keep her seven children alive , and a few months later , she gives birth to premature twins . Two tiny little boys . She cuts the umbilical cord with a stick , and ties it with her own hair . She names the twins after the camp 's commanders to gain their favor , and feeds them with black tea because her milk cannot sustain them . When the soldiers burst in her cell to rape her oldest daughter , she grabs hold of her and refuses to let go , even when they hold a gun to her head . Somehow , the family su
 rvives for 16 months , and then , by extraordinary luck , and the passionate heart of a young American man , Sasha Chanoff , who manages to put her in a U. S. rescue plane , Rose Mapendo and her nine children end up in Phoenix , Arizona , where they 're now living and thriving . Mapendo , in Swahili , means great love . The protagonists of my books are strong and passionate women like Rose Mapendo . I do n't make them up . There 's no need for that . I look around and I see them everywhere . I have worked with women and for women all my life . I know them well . I was born in ancient times , at the end of the world , in a patriarchal Catholic and conservative family . No wonder that by age five I was a raging feminist -- although the term had not reached Chile yet , so nobody knew what the heck was wrong with me . ( Laughter ) I would soon find out that there was a high price to pay for my freedom , and for questioning the patriarchy . But I was happy to pay it , because for every b
 low that I received , I was able to deliver two . ( Laughter ) Once , when my daughter Paula was in her twenties , she said to me that feminism was dated , that I should move on . We had a memorable fight . Feminism is dated ? Yes , for privileged women like my daughter and all of us here today , but not for most of our sisters in the rest of the world who are still forced into premature marriage , prostitution , forced labor -- they have children that they do n't want or they cannot feed . They have no control over their bodies or their lives . They have no education and no freedom . They are raped , beaten up and sometimes killed with impunity , For most Western young women of today , being called a feminist is an insult . Feminism has never been sexy , but let me assure you that it never stopped me from flirting , and I have seldom suffered from lack of men . ( Laughter ) Feminism is not dead , by no means . It has evolved . If you do n't like the term , change it , for Goddess '
  sake . Call it Aphrodite , or Venus , or bimbo , or whatever you want , the name does n't matter , as long as we understand what it is about , and we support it . So here 's another tale of passion , and this is a sad one . The place is a small women 's clinic in a village in Bangladesh . The year is 2005. Jenny is a young American dental hygienist who has gone to the clinic as a volunteer during her three-week vacation . She 's prepared to clean teeth , but when she gets there , she finds out that there are no doctors , no dentists , and the clinic is just a hut full of flies . Outside , there is a line of women who have waited several hours to be treated . The first patient is in excruciating pain because she has several rotten molars . Jenny realizes that the only solution is to pull out the bad teeth . She 's not licensed for that , she has never done it . She risks a lot and she 's terrified . She does n't even have the proper instruments , but fortunately she has brought some
  Novocaine . Jenny has a brave and passionate heart . She murmurs a prayer and she goes ahead with the operation . At the end , the relieved patient kisses her hands . That day the hygienist pulls out many more teeth . The next morning , when she comes again to the so-called clinic , her first patient is waiting for her with her husband . The woman 's face looks like a watermelon . It is so swollen that you ca n't even see the eyes . The husband , furious , threatens to kill the American . Jenny is horrified at what she has done , but then the translator explains that the patient 's condition has nothing to do with the operation . The day before , her husband beat her up because she was not home in time to prepare dinner for him . Millions of women live like this today . They are the poorest of the poor . Although women do two-thirds of the world 's labor , they own less than one percent of the world 's assets . They are paid less than men for the same work if they 're paid at all ,
  and they remain vulnerable because they have no economic independence , and they are constantly threatened by exploitation , violence and abuse . It is a fact that giving women education , work , the ability to control their own income , inherit and own property , benefits the society . If a woman is empowered , her children and her family will be better off . If families prosper , the village prospers , and eventually so does the whole country . Wangari Maathai goes to a village in Kenya . She talks with the women , and explains that the land is barren because they have cut and sold the trees . She gets the women to plant new trees and water them , drop by drop . In a matter of five or six years , they have a forest , the soil is enriched , and the village is saved . The poorest and most backward societies are always those that put women down . Yet this obvious truth is ignored by governments , and also by philanthropy . For every dollar given to a women 's program , 20 dollars ar
 e given to men 's programs . Women are 51 percent of humankind . Empowering them will change everything -- more than technology and design and entertainment . I can promise you that women working together -- linked , informed and educated -- can bring peace and prosperity to this forsaken planet . In any war today , most of the casualties are civilians , mainly women and children . They are collateral damage . Men run the world , and look at the mess we have . What kind of world do we want ? This is a fundamental question that most of us are asking . Does it make sense to participate in the existing world order ? We want a world where life is preserved , and the quality of life is enriched for everybody , not only for the privileged . In January I saw an exhibit of Fernando Botero 's paintings at the UC Berkeley library . No museum or gallery in the United States , except for the New York gallery that carries Botero 's work , has dared to show the paintings because the theme is the 
 Abu Ghraib prison . They are huge paintings of torture and abuse of power , in the voluminous Botero style . I have not been able to get those images out of my mind or my heart . What I fear most is power with impunity . I fear abuse of power , and the power to abuse . In our species , the alpha males define reality , and force the rest of the pack to accept that reality and follow the rules . The rules change all the time , but they always benefit them , and in this case , the trickle-down effect , which does not work in economics , works perfectly . Abuse trickles down from the top of the ladder to the bottom . Women and children , especially the poor , are at the bottom . Even the most destitute of men have someone they can abuse -- a woman or a child . I 'm fed up with the power that a few exert over the many through gender , income , race , and class . I think that the time is ripe to make fundamental changes in our civilization . But for real change , we need feminine energy i
 n the management of the world . We need a critical number of women in positions of power , and we need to nurture the feminine energy in men . I 'm talking about men with young minds , of course . Old guys are hopeless , we have to wait for them to die off . ( Laughter ) Yes , I would love to have Sophia Loren 's long legs and legendary breasts . But given a choice , I would rather have the warrior heart of Wangari Maathai , Somaly Mam , Jenny , and Rose Mapendo . I want to make this world good . Not better , but to make it good . Why not ? It is possible . Look around in this room -- all this knowledge , energy , talent , and technology . Let 's get off our fannies , roll up our sleeves and get to work , passionately , in creating an almost perfect world . Thank you . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/158ted_jessi_arrington_wearing_nothing_new.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/158ted_jessi_arrington_wearing_nothing_new.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/158ted_jessi_arrington_wearing_nothing_new.txt
new file mode 100644
index 0000000..59a4166
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/158ted_jessi_arrington_wearing_nothing_new.txt
@@ -0,0 +1,2 @@
+
+I 'm Jessi , and this is my suitcase . But before I show you what I 've got inside , I 'm going to make a very public confession , and that is , I 'm outfit obsessed . I love finding , wearing , and more recently , photographing and blogging a different colorful , crazy outfit for every single occasion . But I do n't buy anything new . I get all my clothes secondhand from flee markets and thrift stores . Aww , thank you . Secondhand shopping allows me to reduce the impact my wardrobe has on the environment and on my wallet . I get to meet all kinds of great people ; my dollars usually go to a good cause ; I look pretty unique ; and it makes shopping like my own personal treasure hunt . I mean , what am I going to find today ? Is it going to be my size ? Will I like the color ? Will it be under $20 ? If all the answers are yes , I feel as though I 've won . I want to get back to my suitcase and tell you what I packed for this exciting week here at TED . I mean , what does somebody wi
 th all these outfits bring with her ? So I 'm going to show you exactly what I brought . I brought seven pairs of underpants and that 's it . Exactly one week 's worth of undies is all I put in my suitcase . I was betting that I 'd be able to find everything else I could possible want to wear once I got here to Palm Springs . And since you do n't know me as the woman walking around TED in her underwear -- ( Laughter ) that means I found a few things . And I 'd really love to show you my week 's worth of outfits right now . Does that sound good ? ( Applause ) So as I do this , I 'm also going to tell you a few of the life lessons that , believe it or not , I have picked up in these adventures wearing nothing new . So let 's start with Sunday . I call this shiny tiger . You do not have to spend a lot of money to look great . You can almost always look phenomenal for under $50 . This whole outfit , including the jacket , cost me 55 , and it was the most expensive thing that I wore the 
 entire week . Monday : Color is powerful . It is almost physiologically impossible to be in a bad mood when you 're wearing bright red pants . ( Laughter ) If you are happy , you are going to attract other happy people to you . Tuesday : Fitting in is way overrated . I 've spent a whole lot of my life trying to be myself and at the same time fit in . Just be who you are . If you are surrounding yourself with the right people , they will not only get it , they will appreciate it . Wednesday : Embrace your inner child . Sometimes people tell me that I look like I 'm playing dress-up , or that I remind them of their seven year-old . I like to smile and say , " Thank you . " Thursday : Confidence is key . If you think you look good in something , you almost certainly do . And if you do n't think you look good in something , you 're also probably right . I grew up with a mom who taught me this day-in and day-out . But it was n't until I turned 30 that I really got what this meant . And I
  'm going to break it down for you for just a second . If you believe you 're a beautiful person inside and out , there is no look that you ca n't pull off . So there is no excuse for any of us here in this audience . We should be able to rock anything we want to rock . Thank you . ( Applause ) Friday : A universal truth -- five words for you : Gold sequins go with everything . And finally , Saturday : Developing your own unique personal style is a really great way to tell the world something about you without having to say a word . It 's been proven to me time and time again as people have walked up to me this week simply because of what I 'm wearing . And we 've had great conversations . So obviously this is not all going to fit back in my tiny suitcase . So before I go home to Brooklyn , I 'm going to donate everything back . Because the lesson I 'm trying to learn myself this week is that it 's okay to let go . I do n't need to get emotionally attached to these things , because 
 around the corner , there is always going to be another crazy , colorful , shiny outfit just waiting for me , if I put a little love in my heart and look . Thank you very much . ( Applause ) Thank you . ( Applause ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/159ted_john_underkoffler_drive_3d_data_with_a_gesture.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/159ted_john_underkoffler_drive_3d_data_with_a_gesture.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/159ted_john_underkoffler_drive_3d_data_with_a_gesture.txt
new file mode 100644
index 0000000..39990bb
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/159ted_john_underkoffler_drive_3d_data_with_a_gesture.txt
@@ -0,0 +1,2 @@
+
+We 're 25 , 26 years after the advent of the Macintosh , which was an astoundingly seminal event in the history of human-machine interface , and in computation in general . It fundamentally changed the way that people thought about computation , thought about computers , how they used them and who and how many people were able to use them . It was such a radical change , in fact , that the early Macintosh development team in '82 , '83 , '84 , had to write an entirely new operating system from the ground up . Now , this is an interesting little message , and it 's a lesson that has since , I think , been forgotten or lost or something . And that is , namely , that the OS is the interface . The interface is the OS . It 's like the land and the king ( i. e. Arthur ) they 're inseparable , they are one . And to write a new operating system was not a capricious matter . It was n't just a matter of tuning up some graphics routines . There were no graphics routines . There were no mouse dr
 ivers . So it was a necessity . But in the quarter century since then , we 've seen all of the fundamental supporting technologies go berserk . So memory capacity and disk capacity have been multiplied by something between 10,000 and a million . Same thing for processor speeds . Networks , we did n't have networks at all at the time of the Macintosh 's introduction . And that has become the single most salient aspect of how we live with computers . And , of course , graphics : today $84.97 at Best Buy buys you more graphics power than you could have gotten for a million bucks from SGI only a decade ago . So we 've got that incredible ramp-up . Then , on the side , we 've got the Web and , increasingly , the cloud , which is fantastic , but also -- in the regard in which an interface is fundamental -- kind of a distraction . So we 've forgotten to invent new interfaces . Certainly we 've seen , in recent years , a lot of change in the regard . And people are starting to wake up about
  that . So what happens next ? Where do we go from there ? The problem , as we see it , has to do with a single , simple word , " space " or a single , simple phrase , " real world geometry . " Computers and the programming languages that we talk to them in , that we teach them in , are hideously insensate when it comes to space . They do n't understand real world space . It 's a funny thing because the rest of us occupy it quite frequently and quite well . They also do n't understand time , but that 's a matter for a separate talk . So what happens if you start to explain space to them ? One thing you might get is something like the Luminous Room . The Luminous Room is a system in which it 's considered that input and output spaces are co-located . That 's a strangely simple , and yet unexplored idea , right ? When you use a mouse , your hand is down here on the mouse pad . It 's not even on the same plane as what you 're talking about : the pixels are up on the display . So here w
 as a room in which all the walls , floors , ceilings , pets , potted plants , whatever was in there , were capable , not only , of display , but of sensing as well . And that means input and output are in the same space enabling stuff like this . That 's a digital storage in a physical container . The contract is the same as with real word objects in real world containers . Has to come back out , whatever you put in . This little design experiment that was a small office here knew a few other tricks as well . If you presented it with a chess board , it tried to figure out what you might mean by that . And if there was nothing for them to do , the chess pieces eventually got bored and hopped away . The academics who were overseeing this work thought that was too frivolous , so we then built deadly serious applications like this optics prototyping workbench in which a toothpaste cap on a cardboard box becomes a laser . The beam splitters and lenses are represented by physical objects 
 , and the system projects down the laser beam path . So you 've got an interface that has no interface . You operate the world as you operate the real world , which is to say , with your hands . Similarly , a digital wind tunnel with digital wind flowing from right to left . Not that remarkable in a sense ; we did n't invent the mathematics . But if you displayed that on a CRT or flat panel display , it would be meaningless to hold up an arbitrary object , a real world object on it . Here , the real world merges with the simulation . And finally , to pull out all the stops , this is a system called Urp , for urban planners , in which we give architects and urban planners back the models that we confiscated when we insisted that they use CAD systems . And we make the machine meet them half way . Urp projects down digital shadows , as you see here . And if you introduce tools like this inverse clock , then you can control the sun 's position in the sky . That 's 8:00 AM shadows . They
  get a little shorter at 9:00 AM . There you are , swinging the sun around . Short shadows at noon and so forth . And we built up a series of tools like this . There are inter-shadowing studies that children can operate , even though they do n't know anything about urban planning , to move a building , you simply reach out your hand and you move the building . A material wand makes the building into a sort of Frank Gehry thing that reflects light in all directions . Are you blinding passers by and motorists on the freeways ? A zoning tool connects distant structures , a building and a roadway . Are you going to get sued by the zoning commission ? and so forth . Now , if these ideas seem familiar or perhaps even a little dated , that 's great ; they should seem familiar . This work is 15 years old . This stuff was undertaken at MIT and the Media Lab under the incredible direction of Prof. Hiroshi Ishii , director of the Tangible Media Group . And it was that work that was seen by Ale
 x McDowell , one of the world 's legendary production designers . Alex was preparing a little , sort of obscure , indie , arthouse film called " Minority Report " for Steven Spielberg , and invited us to come out from MIT and design the interfaces that would appear in that film . And the great thing about it was that Alex was so dedicated to the idea of verisimilitude , the idea that the putative 2054 that we were painting in the film be believable , that he allowed us to take on that design work as if it were an R&D effort . And the result is sort of gratifyingly perpetual . People still reference those sequences in " Minority Report " when they talk about new UI design . So this led full circle , in a strange way , to build these ideas into what we believe is the necessary future of human machine interface : the Spatial Operating Environment , we call it . So here we have a bunch of stuff , some images . And , using a hand , we can actually exercise six degrees of freedom , six de
 grees of navigational control . And it 's fun to fly through Mr. Beckett 's eye . And you can come back out through the scary orangutan . And that 's all well and good . Let 's do something a little more difficult . Here , we have a whole bunch of disparate images . We can fly around them . So navigation is a fundamental issue . You have to be able to navigate in 3D . Much of what we want computers to help us with in the first place is inherently spatial . And the part that is n't spatial can often be spatialized to allow our wetware to make greater sense of it . Now we can distribute this stuff in many different ways . So we can throw it out like that . Let 's reset it . We can organize it this way . And , of course , it 's not just about navigation , but about manipulation as well . So if we do n't like something , or we 're intensely curious about Ernst Haeckel 's scientific falsifications , we can pull them out like that . And then if it 's time for analysis , we can pull back a
  little bit and ask for a different distribution . Let 's just come down a bit and fly around . So that 's a different way to look at stuff . If you 're of a more analytical nature then you might want , actually , to look at this as a color histogram . So now we 've got the stuff color-sorted , angle maps onto color . And now , if we want to select things , 3D , space , the idea that we 're tracking hands in real space becomes really important because we can reach in , not in 2D , not in fake 2D , but in actual 3D . Here are some selection planes . And we 'll perform this Boolean operation because we really love yellow and tapirs on green grass . So , from there to the world of real work . Here 's a logistics system , a small piece of one that we 're currently building . There 're a lot of elements . And one thing that 's very important is to combine traditional tabular data with three-dimensional and geospatial information . So here 's a familiar place . And we 'll bring this back 
 here for a second . Maybe select a little bit of that . And bring out this graph . And we should , now , be able to fly in here and have a closer look . These are logistics elements that are scattered across the United States . One thing that three-dimensional interactions and the general idea of imbuing computation with space affords you is a final destruction of that unfortunate one to one pairing between human beings and computers . That 's the old way ; that 's the old mantra , one machine , one human , one mouse , one screen . Well , that does n't really cut it anymore . In the real world , we have people who collaborate ; we have people who have to work together . And we have many different displays . And we might want to look at these various images . We might want to ask for some help . The author of this new pointing device is sitting over there , so I can pull it from here to there . These are unrelated machines , right ? So the computation is space soluble and network sol
 uble . So I 'm going to leave that over there because I have a question for Paul . Paul is the designer of the wand , and maybe its easiest for him to come over here and tell me in person what 's going on . So let me get some of these out of the way . Let 's pull this apart : I 'll go ahead and explode it . Kevin , can you help ? Let me see if I can help us find the circuit board . Mind you , it 's a sort of gratuitous field-stripping exercise , but we do it in the lab all the time . All right . So collaborative work , whether it 's immediately co-located or distant and distinct , is always important . And again , that stuff needs to be undertaken in the context of space . And finally , I 'd like to leave you with a glimpse that takes us back to the world of imagery . This is a system called TAMPER , which is a slightly whimsical look at what the future of editing and media manipulation systems might be . We at Oblong believe that media should be accessible in much more fine-grained
  form . So we have a large number of movies stuck inside here . And let 's just pick out a few elements . We can zip through them as a possibility . We can grab elements off the front , where upon they reanimate , come to life , and drag them down onto the table here . We 'll go over to Jacques Tati here and grab our blue friend and put him down on the table as well . We may need more than one . And we probably need , well , we probably need a cowboy to be quite honest . ( Laughter ) Yeah , let 's take that one . ( Laughter ) You see , cowboys and French farce people do n't go well together , and the system knows that . Let me leave with one final thought , and that is that one of the greatest English language writers of the last three decades suggested that great art is always a gift . And he was n't talking about whether the novel costs 24.95 , or whether you have to spring 70 million bucks to buy the stolen Vermeer ; he was talking about the circumstances of its creation and of i
 ts existence . And I think that it 's time that we asked for the same from technology . Technology is capable of expressing and being imbued with a certain generosity , and we need to demand that , in fact . For some of this kind of technology , ground center is a combination of design , which is crucially important . We ca n't have advances in technology any longer unless design is integrated from the very start . And , as well , of efficacy , agency . We 're , as human beings , the creatures that create , and we should make sure our machines aid us in that task and are built in that same image . So I will leave you with that . Thank you . ( Applause ) Chris Anderson : So to ask the obvious question -- actually this is from Bill Gates -- when ? ( John Underkoffler : When ? ) CA : When real ? When for us , not just in a lab and on a stage ? Can it be for every man , or is this just for corporations and movie producers ? JU : No , it has to be for every human being . That 's our goal
  entirely . We wo n't have succeeded unless we take that next big step . I mean it 's been 25 years . Can there really be only one interface ? There ca n't . CA : But does that mean that , at your desk or in your home , you need projectors , cameras ? You know , how can it work ? JU : No , this stuff will be built into the bezel of every display . It 'll be built into architecture . The gloves go away in a matter of months or years . So this is the inevitability about it . CA : So , in your mind , five years time , someone can buy this as part of a standard computer interface ? JU : I think in five years time when you buy a computer , you 'll get this . CA : Well that 's cool . ( Applause ) The world has a habit of surprising us as to how these things are actually used . What do you think , what in your mind is the first killer app for this ? JU : That 's a good question , and we ask ourselves that every day . At the moment , our early-adopter customers -- and these systems are depl
 oyed out in the real world -- do all the big data intensive , data heavy problems with it . So , whether it 's logistics and supply chain management or natural gas and resource extraction , financial services , pharmaceuticals , bioinformatics , those are the topics right now , but that 's not a killer app . And I understand what you 're asking . CA : C'mon , c'mon . Martial arts , games . C'mon . ( Laughter ) John , thank you for making science-fiction real . JU : It 's been a great pleasure . Thank you to you all . ( Applause ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/160ted_joseph_lekuton_tells_a_parable_for_kenya.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/160ted_joseph_lekuton_tells_a_parable_for_kenya.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/160ted_joseph_lekuton_tells_a_parable_for_kenya.txt
new file mode 100644
index 0000000..c533985
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/160ted_joseph_lekuton_tells_a_parable_for_kenya.txt
@@ -0,0 +1,2 @@
+
+My name is Joseph , a Member of Parliament in Kenya . And picture a Masai village . And one evening , government soldiers come , surround the village , and ask each elder to bring one boy to school . That 's how I went to school -- pretty much a government guy pointing a gun , and told my father , " You have to make a choice . " So I walked very comfortably to this missionary school , that was run by an American missionary , and the first thing the American missionary gave me was a candy . I had never in my life ever tasted candy . So I said to myself , with all these hundred other boys , this is where I belong . Stayed -- ( Laughter ) -- when everybody else was dropping out . My family moved . We 're nomads . Every time school closed -- it was a boarding school , and I was seven -- you had to travel until you find them . 50 miles , 40 miles , it does n't matter . You slept in the bush , but you kept going . And I stayed . I do n't know why I stayed , but I stayed . And all of a sud
 den , I passed the national examination , found myself in a very beautiful high school in Kenya . And I finished high school . And just walking , and found a man who gave me a full scholarship to the United States . My mother still lives in a cow-dung hut , none of my brothers are going to school and this man told me , " Here , go . " So I got a scholarship to St. Lawrence University , upstate New York . Finished that , and then after that I went to Harvard Graduate School . Finished that , and then I worked in DC a little bit . I wrote a book for National Geographic , and taught history , US history . And every time I kept going back home , listening to the problems of these people , sick people , people with no water , all the stuff . And every time I go back to America , I kept thinking about them . Then one day , an elder gave me a story , and this story went like this . Long time ago , there was a big war between tribes . And there was this specific tribe that was really afraid
  of this other Luhya tribe . And every time , they sent scouts out there to make sure no one attacked them . So one day , the scouts came running , and told the villagers , " The enemies are coming , only half an hour away -- they 'll be here . " So people scrambled , took their things and ready to go , move out . But there were two men , one man was blind , one man had no legs -- he was born like that . The leader of the chiefs said , " No , sorry , we ca n't take you , you 'll slow us down . We have to flee our women and children , we have to run . " And they were left behind , waiting to die . But these two people worked something out . The blind man said , " Look , I 'm a very strong man , but I ca n't see . " The man with no legs says , " I can see as far as the end of the world , but I ca n't save myself from a cat , or whatever , animals . " So the blind man went down on his knees , down like this , and told the man with no legs to go over his back , and stood up . The man on
  top can see , the blind man can walk . And these guys took off , followed the footsteps of the villagers , until they found and passed them . So this was told to me in a set-up of elders . And it 's a really poor area , I represent Northern Kenya -- most nomadic , remote areas you can even find . And that man told me , " So , here you are , you 've got a good education from America , you have a good life in America , what are you going to do for us ? We want you to be our eyes , we 'll give you the legs . We 'll walk you , you lead us . " So the opportunity came , and I was always thinking about that , what can I do to help my people ? Every time you go to an area where for 43 years of independence , we still do n't have basic health facilities . A man has to be transported in a wheelbarrow to 20 , 30 kilometers for hospital . No clean drinking water . So I said , " I 'm going to dedicate myself , I 'm leaving America . I 'm going to run for office . " So last July -- I moved from 
 America in June , ran in July election and won . And I came for them , and that 's my goal . And right now , I have in place for the last nine months a plan that in five years , every nomad will have clean drinking water . We 're building dispensaries across that constituency . I 'm asking my friends from America to help with bringing nurses or doctors to help us out . I 'm trying to improve infrastructure . I 'm using the knowledge I received from the United States and from my community to move them forward . I 'm trying to develop homegrown solutions to our issues . Because we know , we realize that people outside can come and help us , but if we do n't help ourselves , there 's nothing we can do . So my plan right now , as I continue with introducing students to different fields -- some become doctors , some lawyers -- we want to produce a comprehensive group of people , students , who can come back and help us see a community grow that is in the middle of a huge economic recessi
 on . So as I continue to be a Member of Parliament , and as I continue listening to all of you talking about botany , talking about health , talking about democracy , talking about new inventions , I 'm hoping that one day , in my own little community -- which is 26,000 square kilometers , maybe five times the size of Rhode Island , with no roads -- we 'll be able to become a model to help others develop . Thank you very much . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/161ted_julia_sweeney_has_the_talk.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/161ted_julia_sweeney_has_the_talk.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/161ted_julia_sweeney_has_the_talk.txt
new file mode 100644
index 0000000..dc2bc5b
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/161ted_julia_sweeney_has_the_talk.txt
@@ -0,0 +1,2 @@
+
+I have a daughter , Mulan . And when she was eight , last year , she was doing a report for school , or she had some homework about frogs . And we were at this restaurant . And she said , " So , basically , frogs lay eggs and the eggs turn into tadpoles , and tadpoles turn into frogs . " And I said , " Yeah . You know , I 'm not really up on my frog reproduction that much . It 's the females , I think , that lay the eggs . And then the males fertilize them . And then they become tadpoles and frogs . " And she says , " What ? Only the females have eggs ? " And I said , " Yeah . " And she goes , " And what 's this fertilizing ? " So I kind of said , " Oh , it 's this extra ingredient , you know , that you need to create a new frog from the mom and dad frog . " And she said , " Oh , so is that true for humans too ? " And I thought , " Okay , here we go . " I did n't know it would happen so quick , at eight . I was trying to remember all the guidebooks , and all I could remember was , "
  Only answer the question they 're asking . Do n't give any more information . " So I said , " Yes . " And she said , " And where do , where do human women , where do women lay their eggs ? " And I said , " Well , funny you should ask . We have evolved to have our own pond . We have our very own pond inside our bodies . And we lay our eggs there . We do n't have to worry about other eggs or anything like that . It 's our own pond . And that 's how it happens . " And she goes , " Then how do they get fertilized ? " And I said , " Well , Men , through their penis , they fertilize the eggs by the sperm coming out . And you go through the woman 's vagina . " And so we 're just eating , and her jaw just drops , and she goes , " Mom ! Like where you go to the bathroom ? " And I said , " I know . I know . " ( Laughter ) That 's how we evolved . It does seem odd . It is a little bit like having a waste treatment plant right next to an amusement park . Bad zoning . But ... She 's like , " Wh
 at ? " And she goes , " But Mom , but men and women ca n't ever see each other naked , Mom . So how could that ever happen ? " And then I put my Margaret Mead hat on . " Human males and females develop a special bond , and when they 're much older , much , much older than you , and they have a very special feeling , then they can be naked together . " And she said , " Mom , have you done this before ? " And I said , " Yes . " And she said , " But Mom , you ca n't have kids . " Because she knows that I adopted her and that I ca n't have kids . And I said , " Yes . " And she said , " Well , you do n't have to do that again . " And then I said , " ... " And then she said , " But how does it happen when a man and woman are together ? Like , how do they know that 's the time ? Mom , does the man just say , 'Is now the time to take off my pants ? ' " ( Laughter ) And I said , " Yes . " ( Laughter ) " That is exactly right . That 's exactly how it happens . " So then we 're driving home , 
 and she 's looking out the window , and she goes , " Mom , what if two people just saw each other on the street , like a man and a woman , and they just started doing it . Would that ever happen ? " And I said , " Oh , no . Humans are so private . Oh no . " And then she goes , " What if there was like a party . And there was just like a whole bunch of girls and a whole bunch of boys . And there was a bunch of men and women and they just started doing it , Mom ? Would that ever happen ? " And I said , " Oh , no , no . That 's not how we do it . " Then we got home and we see the cat . And she goes , " Mom , how do cats do it ? " And I go , " Oh , it 's the same . It 's basically the same . " And then she got all caught up in the legs . " But how would the legs go , Mom . I do n't understand the legs . " She goes , " Mom , everyone ca n't do the splits . " And I go , " I know , but the legs ... " I 'm like , " The legs get worked out . " And she goes , " But I just ca n't understand it
  . " So I go , " You know , why do we go on the Internet , and maybe we can see ... " like on Wikipedia . So we go online , and we put in cats mating . And , unfortunately , on Youtube , there 's many cats mating videos . And we watched them , and I 'm so thankful , because she 's just like , " Wow ! This is so amazing . " She goes , " What about dogs ? " So we put in dogs mating , and , you know , we 're watching it , and she 's totally absorbed . And then she goes , " Mom , do you think they would have on the Internet , any humans mating ? " ( Laughter ) And then I realized that I had taken my little eight year-old 's hand , and taken her right into Internet porn . And I looked into this trusting , loving face , and I said , " Oh , no . That would never happen . " Thank you . ( Applause ) Thank you . ( Applause ) Thank you . I 'm so happy to be here . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/162ted_ken_robinson_says_schools_kill_creativity.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/162ted_ken_robinson_says_schools_kill_creativity.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/162ted_ken_robinson_says_schools_kill_creativity.txt
new file mode 100644
index 0000000..97d28e0
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/162ted_ken_robinson_says_schools_kill_creativity.txt
@@ -0,0 +1,2 @@
+
+Good morning . How are you ? It 's been great , has n't it ? I 've been blown away by the whole thing . In fact , I 'm leaving . ( Laughter ) There have been three themes , have n't there , running through the conference , which are relevant to what I want to talk about . One is the extraordinary evidence of human creativity in all of the presentations that we 've had and in all of the people here . Just the variety of it and the range of it . The second is that it 's put us in a place where we have no idea what 's going to happen , in terms of the future . No idea how this may play out . I have an interest in education -- actually , what I find is everybody has an interest in education . Do n't you ? I find this very interesting . If you 're at a dinner party , and you say you work in education -- actually , you 're not often at dinner parties , frankly , if you work in education . ( Laughter ) You 're not asked . And you 're never asked back , curiously . That 's strange to me . B
 ut if you are , and you say to somebody , you know , they say , " What do you do ? " and you say you work in education , you can see the blood run from their face . They 're like , " Oh my God , " you know , " Why me ? My one night out all week . " ( Laughter ) But if you ask about their education , they pin you to the wall . Because it 's one of those things that goes deep with people , am I right ? Like religion , and money and other things . I have a big interest in education , and I think we all do . We have a huge vested interest in it , partly because it 's education that 's meant to take us into this future that we ca n't grasp . If you think of it , children starting school this year will be retiring in 2065. Nobody has a clue -- despite all the expertise that 's been on parade for the past four days -- what the world will look like in five years ' time . And yet we 're meant to be educating them for it . So the unpredictability , I think , is extraordinary . And the third p
 art of this is that we 've all agreed , nonetheless , on the really extraordinary capacities that children have -- their capacities for innovation . I mean , Sirena last night was a marvel , was n't she ? Just seeing what she could do . And she 's exceptional , but I think she 's not , so to speak , exceptional in the whole of childhood . What you have there is a person of extraordinary dedication who found a talent . And my contention is , all kids have tremendous talents . And we squander them , pretty ruthlessly . So I want to talk about education and I want to talk about creativity . My contention is that creativity now is as important in education as literacy , and we should treat it with the same status . ( Applause ) Thank you . That was it , by the way . Thank you very much . ( Laughter ) So , 15 minutes left . Well , I was born ... no . ( Laughter ) I heard a great story recently -- I love telling it -- of a little girl who was in a drawing lesson . She was six and she was 
 at the back , drawing , and the teacher said this little girl hardly ever paid attention , and in this drawing lesson she did . The teacher was fascinated and she went over to her and she said , " What are you drawing ? " And the girl said , " I 'm drawing a picture of God . " And the teacher said , " But nobody knows what God looks like . " And the girl said , " They will in a minute . " ( Laughter ) When my son was four in England -- actually he was four everywhere , to be honest . ( Laughter ) If we 're being strict about it , wherever he went , he was four that year . He was in the Nativity play . Do you remember the story ? No , it was big . It was a big story . Mel Gibson did the sequel . You may have seen it : " Nativity II . " But James got the part of Joseph , which we were thrilled about . We considered this to be one of the lead parts . We had the place crammed full of agents in T-shirts : " James Robinson IS Joseph ! " ( Laughter ) He did n't have to speak , but you know
  the bit where the three kings come in . They come in bearing gifts , and they bring gold , frankincense and myrhh . This really happened . We were sitting there and I think they just went out of sequence , because we talked to the little boy afterward and we said , " You OK with that ? " And he said , " Yeah , why ? Was that wrong ? " They just switched , that was it . Anyway , the three boys came in -- four-year-olds with tea towels on their heads -- and they put these boxes down , and the first boy said , " I bring you gold . " And the second boy said , " I bring you myrhh . " And the third boy said , " Frank sent this . " ( Laughter ) What these things have in common is that kids will take a chance . If they do n't know , they 'll have a go . Am I right ? They 're not frightened of being wrong . Now , I do n't mean to say that being wrong is the same thing as being creative . What we do know is , if you 're not prepared to be wrong , you 'll never come up with anything original 
 . If you 're not prepared to be wrong . And by the time they get to be adults , most kids have lost that capacity . They have become frightened of being wrong . And we run our companies like this , by the way . We stigmatize mistakes . And we 're now running national education systems where mistakes are the worst thing you can make . And the result is that we are educating people out of their creative capacities . Picasso once said this . He said that all children are born artists . The problem is to remain an artist as we grow up . I believe this passionately , that we do n't grow into creativity , we grow out of it . Or rather , we get educated out if it . So why is this ? I lived in Stratford-on-Avon until about five years ago . In fact , we moved from Stratford to Los Angeles . So you can imagine what a seamless transition that was . ( Laughter ) Actually , we lived in a place called Snitterfield , just outside Stratford , which is where Shakespeare 's father was born . Are you 
 struck by a new thought ? I was . You do n't think of Shakespeare having a father , do you ? Do you ? Because you do n't think of Shakespeare being a child , do you ? Shakespeare being seven ? I never thought of it . I mean , he was seven at some point . He was in somebody 's English class , was n't he ? How annoying would that be ? ( Laughter ) " Must try harder . " Being sent to bed by his dad , you know , to Shakespeare , " Go to bed , now , " to William Shakespeare , " and put the pencil down . And stop speaking like that . It 's confusing everybody . " ( Laughter ) Anyway , we moved from Stratford to Los Angeles , and I just want to say a word about the transition , actually . My son did n't want to come . I 've got two kids . He 's 21 now ; my daughter 's 16. He did n't want to come to Los Angeles . He loved it , but he had a girlfriend in England . This was the love of his life , Sarah . He 'd known her for a month . Mind you , they 'd had their fourth anniversary , because i
 t 's a long time when you 're 16. Anyway , he was really upset on the plane , and he said , " I 'll never find another girl like Sarah . " And we were rather pleased about that , frankly , because she was the main reason we were leaving the country . ( Laughter ) But something strikes you when you move to America and when you travel around the world : Every education system on earth has the same hierarchy of subjects . Every one . Does n't matter where you go . You 'd think it would be otherwise , but it is n't . At the top are mathematics and languages , then the humanities , and the bottom are the arts . Everywhere on Earth . And in pretty much every system too , there 's a hierarchy within the arts . Art and music are normally given a higher status in schools than drama and dance . There is n't an education system on the planet that teaches dance every day to children the way we teach them mathematics . Why ? Why not ? I think this is rather important . I think math is very impor
 tant , but so is dance . Children dance all the time if they 're allowed to , we all do . We all have bodies , do n't we ? Did I miss a meeting ? ( Laughter ) Truthfully , what happens is , as children grow up , we start to educate them progressively from the waist up . And then we focus on their heads . And slightly to one side . If you were to visit education , as an alien , and say " What 's it for , public education ? " I think you 'd have to conclude -- if you look at the output , who really succeeds by this , who does everything that they should , who gets all the brownie points , who are the winners -- I think you 'd have to conclude the whole purpose of public education throughout the world is to produce university professors . Is n't it ? They 're the people who come out the top . And I used to be one , so there . ( Laughter ) And I like university professors , but you know , we should n't hold them up as the high-water mark of all human achievement . They 're just a form o
 f life , another form of life . But they 're rather curious , and I say this out of affection for them . There 's something curious about professors in my experience -- not all of them , but typically -- they live in their heads . They live up there , and slightly to one side . They 're disembodied , you know , in a kind of literal way . They look upon their body as a form of transport for their heads , do n't they ? ( Laughter ) It 's a way of getting their head to meetings . If you want real evidence of out-of-body experiences , by the way , get yourself along to a residential conference of senior academics , and pop into the discotheque on the final night . ( Laughter ) And there you will see it -- grown men and women writhing uncontrollably , off the beat , waiting until it ends so they can go home and write a paper about it . Now our education system is predicated on the idea of academic ability . And there 's a reason . The whole system was invented -- around the world , there
  were no public systems of education , really , before the 19th century . They all came into being to meet the needs of industrialism . So the hierarchy is rooted on two ideas . Number one , that the most useful subjects for work are at the top . So you were probably steered benignly away from things at school when you were a kid , things you liked , on the grounds that you would never get a job doing that . Is that right ? Do n't do music , you 're not going to be a musician ; do n't do art , you wo n't be an artist . Benign advice -- now , profoundly mistaken . The whole world is engulfed in a revolution . And the second is academic ability , which has really come to dominate our view of intelligence , because the universities designed the system in their image . If you think of it , the whole system of public education around the world is a protracted process of university entrance . And the consequence is that many highly talented , brilliant , creative people think they 're not
  , because the thing they were good at at school was n't valued , or was actually stigmatized . And I think we ca n't afford to go on that way . In the next 30 years , according to UNESCO , more people worldwide will be graduating through education than since the beginning of history . More people , and it 's the combination of all the things we 've talked about -- technology and its transformation effect on work , and demography and the huge explosion in population . Suddenly , degrees are n't worth anything . Is n't that true ? When I was a student , if you had a degree , you had a job . If you did n't have a job it 's because you did n't want one . And I did n't want one , frankly . ( Laughter ) But now kids with degrees are often heading home to carry on playing video games , because you need an MA where the previous job required a BA , and now you need a PhD for the other . It 's a process of academic inflation . And it indicates the whole structure of education is shifting ben
 eath our feet . We need to radically rethink our view of intelligence . We know three things about intelligence . One , it 's diverse . We think about the world in all the ways that we experience it . We think visually , we think in sound , we think kinesthetically . We think in abstract terms , we think in movement . Secondly , intelligence is dynamic . If you look at the interactions of a human brain , as we heard yesterday from a number of presentations , intelligence is wonderfully interactive . The brain is n't divided into compartments . In fact , creativity -- which I define as the process of having original ideas that have value -- more often than not comes about through the interaction of different disciplinary ways of seeing things . The brain is intentionally -- by the way , there 's a shaft of nerves that joins the two halves of the brain called the corpus callosum . It 's thicker in women . Following off from Helen yesterday , I think this is probably why women are bett
 er at multi-tasking . Because you are , are n't you ? There 's a raft of research , but I know it from my personal life . If my wife is cooking a meal at home -- which is not often , thankfully . ( Laughter ) But you know , she 's doing -- no , she 's good at some things -- but if she 's cooking , you know , she 's dealing with people on the phone , she 's talking to the kids , she 's painting the ceiling , she 's doing open-heart surgery over here . If I 'm cooking , the door is shut , the kids are out , the phone 's on the hook , if she comes in I get annoyed . I say , " Terry , please , I 'm trying to fry an egg in here . Give me a break . " ( Laughter ) Actually , you know that old philosophical thing , if a tree falls in the forest and nobody hears it , did it happen ? Remember that old chestnut ? I saw a great t-shirt really recently which said , " If a man speaks his mind in a forest , and no woman hears him , is he still wrong ? " ( Laughter ) And the third thing about intel
 ligence is , it 's distinct . I 'm doing a new book at the moment called " Epiphany , " which is based on a series of interviews with people about how they discovered their talent . I 'm fascinated by how people got to be there . It 's really prompted by a conversation I had with a wonderful woman who maybe most people have never heard of , she 's called Gillian Lynne , have you heard of her ? Some have . She 's a choreographer and everybody knows her work . She did " Cats , " and " Phantom of the Opera . " She 's wonderful . I used to be on the board of the Royal Ballet , in England , as you can see . Anyway , Gillian and I had lunch one day and I said , " Gillian , how 'd you get to be a dancer ? " And she said it was interesting , when she was at school , she was really hopeless . And the school , in the '30s , wrote to her parents and said , " We think Gillian has a learning disorder . " She could n't concentrate , she was fidgeting . I think now they 'd say she had ADHD . Would
  n't you ? But this was the 1930s , and ADHD had n't been invented at this point . It was n't an available condition . ( Laughter ) People were n't aware they could have that . Anyway , she went to see this specialist . So , this oak-paneled room , and she was there with her mother , and she was led and sat on a chair at the end , and she sat on her hands for 20 minutes while this man talked to her mother about all the problems Gillian was having at school . And at the end of it -- because she was disturbing people , her homework was always late , and so on , little kid of eight -- in the end , the doctor went and sat next to Gillian and said , " Gillian , I 've listened to all these things that your mother 's told me , and I need to speak to her privately . " He said , " Wait here , we 'll be back , we wo n't be very long . " and they went and left her . But as they went out the room , he turned on the radio that was sitting on his desk . And when they got out the room , he said to
  her mother , " Just stand and watch her . " And the minute they left the room , she said , she was on her feet , moving to the music . And they watched for a few minutes and he turned to her mother and said , " Mrs. Lynne , Gillian is n't sick , she 's a dancer . Take her to a dance school . " I said , " What happened ? " She said , " She did . I ca n't tell you how wonderful it was . We walked in this room and it was full of people like me . People who could n't sit still . People who had to move to think . " Who had to move to think . They did ballet , they did tap , they did jazz , they did modern , they did contemporary . She was eventually auditioned for the Royal Ballet School , she became a soloist , she had a wonderful career at the Royal Ballet . She eventually graduated from the Royal Ballet School and founded her own company -- the Gillian Lynne Dance Company -- met Andrew Lloyd Weber . She 's been responsible for some of the most successful musical theater productions i
 n history , she 's given pleasure to millions , and she 's a multi-millionaire . Somebody else might have put her on medication and told her to calm down . Now , I think ... ( Applause ) What I think it comes to is this : Al Gore spoke the other night about ecology , and the revolution that was triggered by Rachel Carson . I believe our only hope for the future is to adopt a new conception of human ecology , one in which we start to reconstitute our conception of the richness of human capacity . Our education system has mined our minds in the way that we strip-mine the earth : for a particular commodity . And for the future , it wo n't serve us . We have to rethink the fundamental principles on which we 're educating our children . There was a wonderful quote by Jonas Salk , who said , " If all the insects were to disappear from the earth , within 50 years all life on Earth would end . If all human beings disappeared from the earth , within 50 years all forms of life would flourish 
 . " And he 's right . What TED celebrates is the gift of the human imagination . We have to be careful now that we use this gift wisely , and that we avert some of the scenarios scenarios that we 've talked about . And the only way we 'll do it is by seeing our creative capacities for the richness they are , and seeing our children for the hope that they are . And our task is to educate their whole being , so they can face this future . By the way -- we may not see this future , but they will . And our job is to help them make something of it . Thank you very much . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/163ted_marcin_jakubowski.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/163ted_marcin_jakubowski.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/163ted_marcin_jakubowski.txt
new file mode 100644
index 0000000..f01af47
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/163ted_marcin_jakubowski.txt
@@ -0,0 +1,2 @@
+
+Hi , my name is Marcin -- farmer , technologist . I was born in Poland , now in the U. S. I started a group called Open Source Ecology . We 've identified the 50 most important machines that we think it takes for modern life to exist -- things from tractors , bread ovens , circuit makers . Then we set out to create an open source , DIY , do it yourself version that anyone can build and maintain at a fraction of the cost . We call this the Global Village Construction Set . So let me tell you a story . So I finished my 20s with a Ph. D. in fusion energy , and I discovered I was useless . I had no practical skills . The world presented me with options , and I took them . I guess you can call it the consumer lifestyle . So I started a farm in Missouri and learned about the economics of farming . I bought a tractor -- then it broke . I paid to get it repaired -- then it broke again . Then pretty soon I was broke too . I realized that the truly appropriate , low-cost tools that I needed t
 o start a sustainable farm and settlement just did n't exist yet . I needed tools that were robust , modular , highly efficient and optimized , low-cost , made from local and recycled materials that would last a lifetime , not designed for obsolescence . I found that I would have to build them myself . So I did just that . And I tested them . And I found that industrial productivity can be achieved on a small scale . So then I published the 3D designs , schematics , instructional videos and budgets on a wiki . Then contributors from all over the world began showing up , prototyping new machines during dedicated project visits . So far , we have prototyped eight of the 50 machines . And now the project is beginning to grow on its own . We know that open source has succeeded with tools for managing knowledge and creativity . And the same is starting to happen with hardware too . We 're focusing on hardware because it is hardware that can change people 's lives in such tangible materia
 l ways . If we can lower the barriers to farming , building , manufacturing , then we can unleash just massive amounts of human potential . That 's not only in the developing world . Our tools are being made for the American farmer , builder , entrepreneur , maker . We 've seen lots of excitement from these people , who can now start a construction business , parts manufacturing , organic CSA or just selling power back to the grid . Our goal is a repository of published designs so clear , so complete , that a single burned DVD is effectively a civilization starter kit . I 've planted a hundred trees in a day . I 've pressed 5,000 bricks in one day from the dirt beneath my feet and built a tractor in six days. From what I 've seen , this is only the beginning . If this idea is truly sound , then the implications are significant . A greater distribution of the means of production , environmentally sound supply chains , and a newly-relevant DIY maker culture can hope to transcend artif
 icial scarcity . We 're exploring the limits of what we all can do to make a better world with open hardware technology . Thank you . ( Applause ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/164ted_mary_roach_10_things_you_didn_t_know_about_orgasm.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/164ted_mary_roach_10_things_you_didn_t_know_about_orgasm.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/164ted_mary_roach_10_things_you_didn_t_know_about_orgasm.txt
new file mode 100644
index 0000000..a48a866
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/164ted_mary_roach_10_things_you_didn_t_know_about_orgasm.txt
@@ -0,0 +1,2 @@
+
+Alright . I 'm going to show you a couple of images from a very diverting paper in The Journal of Ultrasound in Medicine . I 'm going to go way out on a limb and say that it is the most diverting paper ever published in The Journal of Ultrasound in Medicine . The title is " Observations of In-Utero Masturbation . " ( Laughter ) Okay . Now on the left you can see the hand . That 's the big arrow . And the penis on the right . The hand hovering . And over here we have , in the words of radiologist Israel Meisner , " The hand grasping the penis in a fashion resembling masturbation movements . " Bear in mind this was an ultrasound . So it would have been moving images . Orgasm is a reflex of the autonomic nervous system . Now this is the part of the nervous system that deals with the things that we do n't consciously control . Like digestion , heart rate , sexual arousal . And the orgasm reflex can be triggered by a surprisingly broad range of input . Genital stimulation . Duh . But als
 o Kinsey interviewed a woman who could be brought to orgasm by having someone stroke her eyebrow . People with spinal cord injuries , like parapligias , quadriplegias , will often develop a very very sensitive area right above the level of their injury . Wherever that is . There is such a thing as a knee orgasm , in the literature . I think the most curious one that I came across was a case report of a woman who had an orgasm every time she brushed her teeth . ( Laughter ) This was something in the complex sensory-motor action of brushing her teeth was triggering orgasm . And she went to a neurologist who was fascinated . He checked to see if it was something in the toothpaste . But no , it happened with any brand . They stimulated her gums with a toothpick , to see if that was doing it . No. It was the whole , you know , motion . And the amazing thing to me is that now you would think this woman would like have excellent oral hygiene . ( Laughter ) Sadly she -- this is what it said
  in the journal paper -- " She believed that she was possessed by demons and switched to mouthwash for her oral care . " It 's so sad . ( Laughter ) I interviewed , when I was working on the book , I interviewed a woman who can think herself to orgasm . She was part of a study at Rutgers University . You gotta love that . Rutgers . So I interviewed her in Oakland , in a sushi restaurant . And I said , " So , could you do it right here ? " And she said , " Yeah , but I 'd rather finish my meal if you do n't mind . " ( Laughter ) But afterwards she was kind enough to demonstrate on a bench outside . It was remarkable . It took about one minute . And I said to her , " Are you just doing this all the time ? " ( Laughter ) She said , " No. Honestly when I get home I 'm usually too tired . " ( Laughter ) She said that the last time she had done it was on the Disneyland tram . ( Laughter ) The headquarters for orgasm , along the spinal nerve , is something called the sacral nerve root . Wh
 ich is back here . And if you trigger , if you stimulate with an electrode , the precise spot , you will trigger an orgasm . And it is a fact that you can trigger spinal reflexes in dead people . A certain kind of dead person , a beating-heart cadaver . Now this is somebody who is braindead , legally dead , definitely checked out , but is being kept alive on a respirator , so that their organs will be oxygenated for transplantation . Now in one of these braindead people , if you trigger the right spot you will see something every now and then . There is a reflex called the Lazarus reflex . And this is -- I 'll demonstrate as best I can , not being dead . It 's like this . You trigger the spot . The dead guy , or gal , goes ... like that . Very unsettling for people working in pathology labs . ( Laughter ) Now if you can trigger the Lazarus reflex in a dead person , why not the orgasm reflex ? I asked this question to a brain death expert , Stephanie Mann , who was foolish enough to 
 return my emails . ( Laughter ) I said , " So , could you concievably trigger an orgasm in a dead person ? " She said , " Yes , if the sacral nerve is being oxygenated . You conceivably could . " Obviously it would n't be as much fun for the person . But it would be an orgasm -- ( Laughter ) nonetheless . I actually suggested to -- there is a researcher at the University of Alabama who does orgasm research . I said to her , " You should do an experiment . You know ? You can get cadavers if you work at a university . " I said , " You should actually do this . " She said , " You get the human subjects review board approval for this one . " ( Laughter ) According to 1930s marriage manual author , Theodoor Van de Velde , a slight seminal odor can be detected on the breath of a woman within about an hour after sexual intercourse . Theodore Van de Velde was something of a semen connoisseur . ( Laughter ) This is a guy writing a book , " Ideal Marriage , " you know . Very heavy hetero guy 
 . But he wrote in this book , " Ideal Marriage , " he said that he could differentiate between the semen of a young man , which he said had a fresh , exhilarating smell , and the semen of mature men , whose semen smelled quote , " Remarkably like that of the flowers of the Spanish chestnut . Sometimes quite freshly floral , and then again sometimes extremely pungent . " ( Laughter ) Okay . 1999 , in the state of Israel , a man began hiccuping . And this was one of those cases that went on and on . He tried everything his friends suggested . Nothing seemed to help . Days went by . At a certain point , the man , still hiccuping , had sex with his wife . And lo and behold , the hiccups went away . He told his doctor , who published a case report in a Canadian medical journal under the title , " Sexual Intercourse as a Potential Treatment for Intractable Hiccups . " I love this article because at a certain point they suggested that unattached hiccupers could try masturbation . ( Laughte
 r ) I love that because there is like a whole demographic . Unattached hiccupers . ( Laughter ) Married . Single . Unattached hiccuper . In the 1900s , early 1900s gynecologists , a lot of gynecologists believed that when a woman has an orgasm the contractions serve to suck the semen up through the cervix and sort of deliver it really quickly to the egg . Thereby upping the odds of conception . It was called the " upsuck " theory . ( Laughter ) If you go all the way back to Hippocrates , physicians believed that orgasm in women was not just helpful for conception , but necessary . Doctors back then were routinely telling men the importance of pleasuring their wives . Marriage manual author and semen sniffer Theodore Van de Velde -- ( Laughter ) has a line in his book . I loved this guy . I got a lot of mileage out of Theodore Van de Velde . He had this line in his book , that supposedly comes from the Habsburg Monarchy . Where there was an empress Maria Theresa , who was having trou
 ble conceiving . And apparently the royal court physician said to her , " I am of the opinion that the vulva of your most sacred majesty be titillated for some time prior to intercourse . " ( Laughter ) It 's apparently , I do n't know , on the record somewhere . Masters and Johnson : now we 're moving forward to the 1950s . Masters and Johnson were upsuck skeptics . Which is also really fun to say . They did n't buy it . And they decided , being Masters and Johnson , that they would get to the bottom of it . They brought women into the lab . I think it was five women . And outfitted them with cervical caps containing artificial semen . And in the artificial semen was a radio-opaque substance , such that it would show up on an X-ray . This is the 1950s . Anyway these women sat in front of an X-ray device . And they masturbated . And Masters and Johnson looked to see if the semen was being sucked up . Did not find any evidence of upsuck . You may be wondering , " How do you make arti
 ficial semen ? " ( Laughter ) I have an answer for you . I have two answers . You can use flour and water , or cornstarch and water . I actually found three separate recipes in the literature . ( Laughter ) My favorite being the one that says -- you know , they have the ingredients listed , and then in a recipe it will say , for example , " Yield : two dozen cupcakes . " This one said , " Yield : one ejaculate . " ( Laughter ) There 's another way that orgasm might boost fertility . This one involves men . Sperm that sit around in the body for a week or more start to develop abnormalities that make them less effective at head banging their way into the egg . British sexologist Roy Levin has speculated that this is perhaps why men evolved to be such enthusiastic and frequent masturbators . He said , " If I keep tossing myself off I get fresh sperm being made . " Which I thought was an interesting idea , theory . So now you have an evolutionary excuse . ( Laughter ) Okay . ( Laughter 
 ) Alrighty . There is considerable evidence for upsuck in the animal kingdom . Pigs , for instance . In Denmark , the Danish National Committee for Pig Production found out that if you sexually stimulate a sow while you artificially inseminate her , you will see a six-percent increase in the farrowing rate , which is the number of piglets produced . So they came up with this plan . This five-point stimulation plan for the sows . And they had the farmers -- there is posters they put in the barn , and they have a DVD . And I got a copy of this DVD . ( Laughter ) This is my unveiling . Because I am going to show you a clip . ( Laughter ) So uh , okay . Now here we go in to the -- la la la , off to work . It all looks very innocent . He 's going to be doing things with his hands that the boar would use his snout , lacking hands . Okay . ( Laughter ) This is it . The boar has a very odd courtship repertoire . ( Laughter ) This is to mimic the weight of the boar . ( Laughter ) You should 
 know , the clitoris of the pig , inside the vagina . So this may be sort of titillating for her . Here we go . ( Laughter ) And the happy result . ( Applause ) I love this video . There is a point in this video , towards the beginning where they zoom in for a close up of his hand with his wedding ring , as if to say , " It 's okay , it 's just his job . He really does like women . " ( Laughter ) Okay . Now I said -- when I was in Denmark , my host was named Anne Marie . And I said , " So why do n't you just stimulate the clitoris of the pig ? Why do n't you have the farmers do that ? That 's not one of your five steps . " She said -- I have to read you what she said , because I love it . She said , " It was a big hurdle just to get farmers to touch underneath the vulva . So we thought let 's not mention the clitoris right now . " ( Laughter ) Shy but ambitious pig farmers , however , can purchase a -- this is true -- a sow vibrator , that hangs on the sperm feeder tube to vibrate . 
 Because , as I mentioned , the clitoris is inside the vagina . So possibly , you know , a little more arousing than it looks . And I also said to her , " Now these sows . I mean , you may have noticed there , The sow does n't look to be in the throes of ecstasy . " And she said , " You ca n't make that conclusion . " Because animals do n't register pain or pleasure on their faces , in the same way that we do . They tend to -- pigs , for example , are more like dogs . They use the upper half of the face . The ears are very expressive . So you 're not really sure what 's going on with the pig . Primates , on the other hand , we use our mouths more . This is the ejaculation face of the stump-tailed macaque . ( Laughter ) And , interestingly , this has been observed in female macaques . But only when mounting another female . ( Laughter ) Masters and Johnson , in the 1950s , they decided , okay , we 're going to figure out the entire human sexual response cycle . From arousal , all the 
 way through orgasm , in men and women . Everything that happens in the human body . Okay , with women , a lot of this is happening inside . This did not stop Masters and Johnson . They developed an artificial coition machine . This is basically a penis camera on a motor . There is a phallus , clear acrylic phallus , with a camera and a light source , attached to a motor that is kind of going like this . And the woman would have sex with it . That is what they would do . Pretty amazing . Sadly , this device has been dismantled . This just kills me . Not because I wanted to use it . I wanted to see it . ( Laughter ) One fine day Alfred Kinsey decided to calculate the average distance traveled by ejaculated semen . This was not idle curiosity . Doctor Kinsey had heard -- and there was a theory kind of going around at the time , this being the 1940s , that the force with which semen is thrown against the cervix was a factor in fertility . Kinsey thought it was bunk . So he got to work .
  He got together in his lab 300 men , a measuring tape , and a movie camera . ( Laughter ) And in fact he found that in three quarters of the men the stuff just kind of slopped out . It was n't spurted or thrown or ejected under great force . However , the record holder landed just shy of the eight foot mark . Which is impressive . ( Laughter ) ( Applause ) Yes . Exactly . ( Laughter ) Sadly , he 's anonymous . His name is not mentioned . In his write up , in his write up of this experiment in his book , Kinsey wrote , " Two sheets were laid down to protect the oriental carpets . " ( Laughter ) Which is my second favorite line in the entire ouevre of Alfred Kinsey . My favorite being , " Cheese crumbs spread before a pair of copulating rats will distract the female , but not the male . " ( Laughter ) Thank you very much . ( Applause ) Thanks ! 
\ No newline at end of file


[31/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/65FictPoeE_Usher_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/65FictPoeE_Usher_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/65FictPoeE_Usher_EN.txt.txt
new file mode 100644
index 0000000..9092a8a
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/65FictPoeE_Usher_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff THE FALL OF THE HOUSE OF USHER -------------------------------------------------------------------------------- Son c\u0153ur est un luth suspendu ; Sit�t qu\u2019on le touche il r�sonne . De B�ranger . During the whole of a dull , dark , and soundless day in the autumn of the year , when the clouds hung oppressively low in the heavens , I had been passing alone , on horseback , through a singularly dreary tract of country ; and at length found myself , as the shades of the evening drew on , within view of the melancholy House of Usher . I know not how it was\u2014but , with the first glimpse of the building , a sense of insufferable gloom pervaded my spirit . I say insufferable ; for the feeling was unrelieved by any of that half-pleasurable , because poetic , sentiment , with which the mind usually receives even the sternest natural images of the desolate or terrible . I looked upon the scene before me\u2014upon the mere house , and the simple landscape features of the domain\u2014upon the
  bleak walls\u2014upon the vacant eye-like windows\u2014upon a few rank sedges\u2014and upon a few white trunks of decayed trees\u2014with an utter depression of soul which I can compare to no earthly sensation more properly than to the after-dream of the reveler upon opium\u2014the bitter lapse into every-day life\u2014the hideous dropping off of the veil . There was an iciness , a sinking , a sickening of the heart\u2014an unredeemed dreariness of thought which no goading of the imagination could torture into aught of the sublime . What was it\u2014I paused to think\u2014what was it that so unnerved me in the contemplation of the House of Usher ? It was a mystery all insoluble ; nor could I grapple with the shadowy fancies that crowded upon me as I pondered . I was forced to fall back upon the unsatisfactory conclusion that while , beyond doubt , there are combinations of very simple natural objects which have the power of thus affecting us , still the analysis of this power lies among considerations beyond
  our depth . It was possible , I reflected , that a mere different arrangement of the particulars of the scene , of the details of the picture , would be sufficient to modify , or perhaps to annihilate its capacity for sorrowful impression ; and , acting upon this idea , I reined my horse to the precipitous brink of a black and lurid tarn that lay in unruffled luster by the dwelling , and gazed down\u2014but with a shudder even more thrilling than before\u2014upon the remodeled and inverted images of the gray sedge , and the ghastly tree stems , and the vacant and eye-like windows . Nevertheless , in this mansion of gloom I now proposed to myself a sojourn of some weeks . Its proprietor , Roderick Usher , had been one of my boon companions in boyhood ; but many years had elapsed since our last meeting . A letter , however , had lately reached me in a distant part of the country\u2014a letter from him\u2014which , in its wildly importunate nature , had admitted of no other than a personal reply 
 . The MS . gave evidence of nervous agitation . The writer spoke of acute bodily illness\u2014of a mental disorder which oppressed him , and of an earnest desire to see me , as his best , and indeed his only personal friend , with a view of attempting , by the cheerfulness of my society , some alleviation of his malady . It was the manner in which all this , and much more , was said\u2014it was the apparent heart that went with his request\u2014which allowed me no room for hesitation ; and I accordingly obeyed forthwith what I still considered a very singular summons . Although , as boys , we had been even intimate associates , yet I really knew little of my friend . His reserve had been always excessive and habitual . I was aware , however , that his very ancient family had been noted , time out of mind , for a peculiar sensibility of temperament , displaying itself , through long ages , in many works of exalted art , and manifested , of late , in repeated deeds of munificent , yet unobtrus
 ive charity , as well as in a passionate devotion to the intricacies , perhaps even more than to the orthodox and easily recognizable beauties , of musical science . I had learned , too , the very remarkable fact that the stem of the Usher race , all time-honored as it was , had put forth , at no period , any enduring branch ; in other words , that the entire family lay in the direct line of descent , and had always , with very trifling and very temporary variation , so lain . It was this deficiency , I considered , while running over in thought the perfect keeping of the character of the premises with the accredited character of the people , and while speculating upon the possible influence which the one , in the long lapse of centuries , might have exercised upon the other\u2014it was this deficiency , perhaps , of collateral issue , and the consequent undeviating transmission , from sire to son , of the patrimony with the name , which had , at length , so identified the two as to me
 rge the original title of the estate in the quaint and equivocal appellation of the " House of Usher"\u2014an appellation which seemed to include , in the minds of the peasantry who used it , both the family and the family mansion . I have said that the sole effect of my somewhat childish experiment of looking down within the tarn had been to deepen the first singular impression . There can be no doubt that the consciousness of the rapid increase of my superstition\u2014for why should I not so term it ? \u2014 served mainly to accelerate the increase itself . Such , I have long known , is the paradoxical law of all sentiments having terror as a basis . And it might have been for this reason only , that , when I again uplifted my eyes to the house itself , from its image in the pool , there grew in my mind a strange fancy\u2014a fancy so ridiculous , indeed , that I but mention it to show the vivid force of the sensations which oppressed me . I had so worked upon my imagination as really to beli
 eve that about the whole mansion and domain there hung an atmosphere peculiar to themselves and their immediate vicinity\u2014an atmosphere which had no affinity with the air of heaven , but which had reeked up from the decayed trees , and the gray wall , and the silent tarn\u2014a pestilent and mystic vapor , dull , sluggish , faintly discernible , and leaden-hued . Shaking off from my spirit what must have been a dream , I scanned more narrowly the real aspect of the building . Its principal feature seemed to be that of an excessive antiquity . The discoloration of ages had been great . Minute fungi overspread the whole exterior , hanging in a fine , tangled web-work from the eaves . Yet all this was apart from any extraordinary dilapidation . No portion of the masonry had fallen ; and there appeared to be a wild inconsistency between its still perfect adaptation of parts , and the crumbling condition of the individual stones . In this there was much that reminded me of the specious tot
 ality of old woodwork which has rotted for years in some neglected vault , with no disturbance from the breath of the external air . Beyond this indication of extensive decay , however , the fabric gave little token of instability . Perhaps the eye of a scrutinizing observer might have discovered a barely perceptible fissure , which , extending from the roof of the building in front , made its way down the wall in a zigzag direction , until it became lost in the sullen waters of the tarn . Noticing these things , I rode over a short causeway to the house . A servant in waiting took my horse , and I entered the Gothic archway of the hall . A valet , of stealthy step , thence conducted me , in silence , through many dark and intricate passages in my progress to the studio of his master . Much that I encountered on the way contributed , I know not how , to heighten the vague sentiments of which I have already spoken . While the objects around me\u2014while the carvings of the ceilings , t
 he somber tapestries of the walls , the ebon blackness of the floors , and the phantasmagoric armorial trophies which rattled as I strode , were but matters to which , or to such as which , I had been accustomed from my infancy\u2014while I hesitated not to acknowledge how familiar was all this\u2014I still wondered to find how unfamiliar were the fancies which ordinary images were stirring up . On one of the staircases I met the physician of the family . His countenance , I thought , wore a mingled expression of low cunning and perplexity . He accosted me with trepidation and passed on . The valet now threw open a door and ushered me into the presence of his master . The room in which I found myself was very large and lofty . The windows were long , narrow , and pointed , and at so vast a distance from the black oaken floor as to be altogether inaccessible from within . Feeble gleams of encrimsoned light made their way through the trellised panes , and served to render sufficiently disti
 nct the more prominent objects around ; the eye , however , struggled in vain to reach the remoter angles of the chamber , or the recesses of the vaulted and fretted ceiling . Dark draperies hung upon the walls . The general furniture was profuse , comfortless , antique , and tattered . Many books and musical instruments lay scattered about , but failed to give any vitality to the scene . I felt that I breathed an atmosphere of sorrow . An air of stern , deep , and irredeemable gloom hung over and pervaded all . Upon my entrance , Usher arose from a sofa on which he had been lying at full length , and greeted me with a vivacious warmth which had much in it , I at first thought , of an overdone cordiality\u2014of the constrained effort of the ennuy� man of the world . A glance , however , at his countenance convinced me of his perfect sincerity . We sat down ; and for some moments , while he spoke not , I gazed upon him with a feeling half of pity , half of awe . Surely , man had never
  before so terribly altered , in so brief a period , as had Roderick Usher ! It was with difficulty that I could bring myself to admit the identity of the wan being before me with the companion of my early boyhood . Yet the character of his face had been at all times remarkable . A cadaverousness of complexion ; an eye large , liquid , and luminous beyond comparison ; lips somewhat thin and very pallid , but of a surpassingly beautiful curve ; a nose of a delicate Hebrew model , but with a breadth of nostril unusual in similar formations ; a finely molded chin , speaking , in its want of prominence , of a want of moral energy ; hair of a more than web-like softness and tenuity ; these features , with an inordinate expansion above the regions of the temple , made up altogether a countenance not easily to be forgotten . And now in the mere exaggeration of the prevailing character of these features , and of the expression they were wont to convey , lay so much of change that I doubted 
 to whom I spoke . The now ghastly pallor of the skin , and the now miraculous luster of the eye , above all things startled and even awed me . The silken hair , too , had been suffered to grow all unheeded , and as , in its wild gossamer texture , it floated rather than fell about the face , I could not , even with effort , connect its arabesque expression with any idea of simple humanity . In the manner of my friend I was at once struck with an incoherence\u2014an inconsistency ; and I soon found this to arise from a series of feeble and futile struggles to overcome an habitual trepidancy , an excessive nervous agitation . For something of this nature I had indeed been prepared , no less by his letter than by reminiscences of certain boyish traits , and by conclusions deduced from his peculiar physical conformation and temperament . His action was alternately vivacious and sullen . His voice varied rapidly from a tremulous indecision ( when the animal spirits seemed utterly in abeyanc
 e ) to that species of energetic concision\u2014that abrupt , weighty , unhurried , and hollow-sounding enunciation\u2014that leaden , self-balanced , and perfectly modulated guttural utterance , which may be observed in the lost drunkard , or the irreclaimable eater of opium , during the periods of his most intense excitement . It was thus that he spoke of the object of my visit , of his earnest desire to see me , and of the solace he expected me to afford him . He entered , at some length , into what he conceived to be the nature of his malady . It was , he said , a constitutional and a family evil , and one for which he despaired to find a remedy\u2014a mere nervous affection , he immediately added , which would undoubtedly soon pass off . It displayed itself in a host of unnatural sensations . Some of these , as he detailed them , interested and bewildered me ; although , perhaps , the terms and the general manner of the narration had their weight . He suffered much from a morbid acutene
 ss of the senses . The most insipid food was alone endurable ; he could wear only garments of certain texture ; the odors of all flowers were oppressive ; his eyes were tortured by even a faint light ; and there were but peculiar sounds , and these from stringed instruments , which did not inspire him with horror . To an anomalous species of terror I found him a bounden slave . " I shall perish , " said he , " I must perish in this deplorable folly . Thus , thus , and not otherwise , shall I be lost . I dread the events of the future , not in themselves , but in their results . I shudder at the thought of any , even the most trivial , incident , which may operate upon this intolerable agitation of soul . I have , indeed , no abhorrence of danger , except in its absolute effect\u2014in terror . In this unnerved\u2014in this pitiable condition\u2014I feel that the period will sooner or later arrive when I must abandon life and reason together , in some struggle with the grim phantasm , Fear . 
 " I learned , moreover , at intervals , and through broken and equivocal hints , another singular feature of his mental condition . He was enchained by certain superstitious impressions in regard to the dwelling which he tenanted , and whence , for many years , he had never ventured forth\u2014in regard to an influence whose supposititious force was conveyed in terms too shadowy here to be restated\u2014an influence which some peculiarities in the mere form and substance of his family mansion , had , by dint of long sufferance , he said , obtained over his spirit\u2014an effect which the physique of the gray walls and turrets , and of the dim tarn into which they all looked down , had , at length , brought about upon the morale of his existence . He admitted , however , although with hesitation , that much of the peculiar gloom which thus afflicted him could be traced to a more natural and far more palpable origin\u2014to the severe and long-continued illness\u2014indeed to the evidently approachi
 ng dissolution\u2014of a tenderly beloved sister , his sole companion for long years , his last and only relative on earth . " Her decease , " he said , with a bitterness which I can never forget , " would leave him ( him the hopeless and the frail ) the last of the ancient race of the Ushers . " While he spoke , the lady Madeline ( for so was she called ) passed slowly through a remote portion of the apartment , and , without having noticed my presence , disappeared . I regarded her with an utter astonishment not unmingled with dread ; and yet I found it impossible to account for such feelings . A sensation of stupor oppressed me , as my eyes followed her retreating steps . When a door , at length , closed upon her , my glance sought instinctively and eagerly the countenance of the brother ; but he had buried his face in his hands , and I could only perceive that a far more than ordinary wanness had overspread the emaciated fingers through which trickled many passionate tears . The di
 sease of the lady Madeline had long baffled the skill of her physicians . A settled apathy , a gradual wasting away of the person , and frequent although transient affections of a partially cataleptical character , were the unusual diagnosis . Hitherto she had steadily borne up against the pressure of her malady , and had not betaken herself finally to bed ; but on the closing in of the evening of my arrival at the house , she succumbed ( as her brother told me at night with inexpressible agitation ) to the prostrating power of the destroyer ; and I learned that the glimpse I had obtained of her person would thus probably be the last I should obtain\u2014that the lady , at least while living , would be seen by me no more . For several days ensuing her name was unmentioned by either Usher or myself ; and during this period I was busied in earnest endeavors to alleviate the melancholy of my friend . We painted and read together ; or I listened , as if in a dream , to the wild improvisati
 ons of his speaking guitar . And thus , as a closer and still closer intimacy admitted me more unreservedly into the recesses of his spirit , the more bitterly did I perceive the futility of all attempt at cheering a mind from which darkness , as if an inherent positive quality , poured forth upon all objects of the moral and physical universe , in one unceasing radiation of gloom . I shall ever bear about me a memory of the many solemn hours I thus spent alone with the master of the House of Usher . Yet I should fail in any attempt to convey an idea of the exact character of the studies , or of the occupations in which he involved me , or led me the way . An excited and highly distempered ideality threw a sulphurous luster over all . His long , improvised dirges will ring forever in my ears . Among other things , I hold painfully in mind a certain singular perversion and amplification of the wild air of the last waltz of Von Weber . From the paintings over which his elaborate fancy
  brooded , and which grew , touch by touch , into vaguenesses at which I shuddered the more thrillingly because I shuddered knowing not why , \u2014 from these paintings ( vivid as their images now are before me ) I would in vain endeavor to deduce more than a small portion which should lie within the compass of merely written words . By the utter simplicity , by the nakedness of his designs , he arrested and overawed attention . If ever mortal painted an idea , that mortal was Roderick Usher . For me , at least\u2014in the circumstances then surrounding me\u2014there arose out of the pure abstractions which the hypochondriac contrived to throw upon his canvas , an intensity of intolerable awe , no shadow of which felt I ever yet in the contemplation of the certainly glowing yet too concrete reveries of Fuseli . One of the phantasmagoric conceptions of my friend , partaking not so rigidly of the spirit of abstraction , may be shadowed forth , although feebly , in words . A small picture pres
 ented the interior of an immensely long and rectangular vault or tunnel , with low walls , smooth , white , and without interruption or device . Certain accessory points of the design served well to convey the idea that this excavation lay at an exceeding depth below the surface of the earth . No outlet was observed in any portion of its vast extent , and no torch or other artificial source of light was discernible ; yet a flood of intense rays rolled throughout , and bathed the whole in a ghastly and inappropriate splendor . I have just spoken of that morbid condition of the auditory nerve which rendered all music intolerable to the sufferer , with the exception of certain effects of stringed instruments . It was , perhaps , the narrow limits to which he thus confined himself upon the guitar , which gave birth , in great measure , to the fantastic character of his performances . But the fervid facility of his impromptus could not be so accounted for . They must have been , and were
  , in the notes , as well as in the words of his wild fantasias ( for he not unfrequently accompanied himself with rimed verbal improvisations ) , the result of that intense mental collectedness and concentration to which I have previously alluded as observable only in particular moments of the highest artificial excitement . The words of one of these rhapsodies I have easily remembered . I was , perhaps , the more forcibly impressed with it , as he gave it , because , in the under or mystic current of its meaning , I fancied that I perceived , and for the first time , a full consciousness on the part of Usher , of the tottering of his lofty reason upon her throne . The verses , which were entitled " The Haunted Palace , " ran very nearly , if not accurately , thus : I. In the greenest of our valleys , By good angels tenanted , Once a fair and stately palace \u2014 Radiant palace\u2014reared its head . In the monarch Thought 's dominion \u2014 It stood there ! Never seraph spread a pinion Ov
 er fabric half so fair . II . Banners yellow , glorious , golden , On its roof did float and flow ; ( This\u2014all this\u2014was in the olden Time long ago ) And every gentle air that dallied , In that sweet day , Along the ramparts plumed and pallid , A wing�d odor went away . III . Wanderers in that happy valley Through two luminous windows saw Spirits moving musically To a lute 's well-tun�d law , Round about a throne , where sitting ( Porphyrogene ! ) In state his glory well befitting , The ruler of the realm was seen . IV . And all with pearl and ruby glowing Was the fair palace door , Through which came flowing , flowing , flowing And sparkling evermore , A troop of Echoes whose sweet duty Was but to sing , In voices of surpassing beauty , The wit and wisdom of their king . V. But evil things , in robes of sorrow , Assailed the monarch 's high estate ( Ah , let us mourn , for never morrow Shall dawn upon him , desolate ! ) ; And , round about his home , the glory That blushed and
  bloomed Is but a dim-remembered story Of the old time entombed . VI . And travelers now within that valley , Through the red-litten windows , see Vast forms that move fantastically To a discordant melody ; While , like a rapid ghastly river , Through the pale door , A hideous throng rush out forever , And laugh\u2014but smile no more . I well remember that suggestions arising from this ballad led us into a train of thought wherein there became manifest an opinion of Usher 's which I mention not so much on account of its novelty ( for other men[1 ] have thought thus ) as on account of the pertinacity with which he maintained it . This opinion , in its general form , was that of the sentience of all vegetable things . But , in his disordered fancy , the idea had assumed a more daring character , and trespassed , under certain conditions , upon the kingdom of inorganization . I lack words to express the full extent or the earnest abandon of his persuasion . The belief , however , was con
 nected ( as I have previously hinted ) with the gray stones of the home of his forefathers . The conditions of the sentience had been here , he imagined , fulfilled in the method of collocation of these stones\u2014in the order of their arrangement , as well as in that of the many fungi which overspread them , and of the decayed trees which stood around\u2014above all , in the long-undisturbed endurance of this arrangement , and in its reduplication in the still waters of the tarn . Its evidence\u2014the evidence of the sentience\u2014was to be seen , he said ( and I here started as he spoke ) , in the gradual yet certain condensation of an atmosphere of their own about the waters and the walls . The result was discoverable , he added , in that silent , yet importunate and terrible influence which for centuries had molded the destinies of his family , and which made him what I now saw him\u2014what he was . Such opinions need no comment , and I will make none . Our books\u2014the books which , for ye
 ars , had formed no small portion of the mental existence of the invalid\u2014were , as might be supposed , in strict keeping with this character of phantasm . We pored together over such works as the Ververt et Chartreuse of Gresset ; the Belphegor of Machiavelli ; the Heaven and Hell of Swedenborg ; the Subterranean Voyage of Nicholas Klimm by Holberg ; the Chiromancy of Robert Flud , of Jean D\u2019Indagin� , and of De la Chambre ; the Journey into the Blue Distance of Tieck ; and the City of the Sun of Campanella . One favorite volume was a small octavo edition of the Directorium Inquisitorium , by the Dominican Eymeric de Cironne ; and there were passages in Pomponius Mela , about the old African Satyrs and \u0152gipans , over which Usher would sit dreaming for hours . His chief delight , however , was found in the perusal of an exceedingly rare and curious book in quarto Gothic\u2014the manual of a forgotten church\u2014the Vigiliae Mortuorum secundum Chorum Ecclesiae Maguntinae . I could no
 t help thinking of the wild ritual of this work , and of its probable influence upon the hypochondriac , when , one evening , having informed me abruptly that the lady Madeline was no more , he stated his intention of preserving her corpse for a fortnight ( previously to its final interment ) in one of the numerous vaults within the main walls of the building . The worldly reason , however , assigned for this singular proceeding , was one which I did not feel at liberty to dispute . The brother had been led to his resolution , so he told me , by consideration of the unusual character of the malady of the deceased , of certain obtrusive and eager inquiries on the part of her medical men , and of the remote and exposed situation of the burial ground of the family . I will not deny that when I called to mind the sinister countenance of the person whom I met upon the staircase , on the day of my arrival at the house , I had no desire to oppose what I regarded as at best but a harmless ,
  and by no means an unnatural precaution . At the request of Usher , I personally aided him in the arrangements for the temporary entombment . The body having been encoffined , we two alone bore it to its rest . The vault in which we placed it ( and which had been so long unopened that our torches , half smothered in its oppressive atmosphere , gave us little opportunity for investigation ) was small , damp , and entirely without means of admission for light ; lying , at great depth , immediately beneath that portion of the building in which was my own sleeping apartment . It had been used , apparently , in remote feudal times , for the worst purposes of a donjon-keep , and in later days , as a place of deposit for powder , or some other highly combustible substance , as a portion of its floor , and the whole interior of a long archway through which we reached it , were carefully sheathed with copper . The door , of massive iron , had been also similarly protected . Its immense weig
 ht caused an unusually sharp grating sound , as it moved upon its hinges . Having deposited our mournful burden upon tressels within this region of horror , we partially turned aside the yet unscrewed lid of the coffin , and looked upon the face of the tenant . A striking similitude between the brother and sister now first arrested my attention ; and Usher , divining , perhaps , my thoughts , murmured out some few words from which I learned that the deceased and himself had been twins , and that sympathies of a scarcely intelligible nature had always existed between them . Our glances , however , rested not long upon the dead\u2014for we could not regard her unawed . The disease which had thus entombed the lady in the maturity of youth , had left , as usual in all maladies of a strictly cataleptical character , the mockery of a faint blush upon the bosom and the face , and that suspiciously lingering smile upon the lip which is so terrible in death . We replaced and screwed down the li
 d , and having secured the door of iron , made our way , with toil , into the scarcely less gloomy apartments of the upper portion of the house . And now , some days of bitter grief having elapsed , an observable change came over the features of the mental disorder of my friend . His ordinary manner had vanished . His ordinary occupations were neglected or forgotten . He roamed from chamber to chamber with hurried , unequal , and objectless step . The pallor of his countenance had assumed , if possible , a more ghastly hue\u2014but the luminousness of his eye had utterly gone out . The once occasional huskiness of his tone was heard no more ; and a tremulous quaver , as if of extreme terror , habitually characterized his utterance . There were times , indeed , when I thought his unceasingly agitated mind was laboring with some oppressive secret , to divulge which he struggled for the necessary courage . At times , again , I was obliged to resolve all into the mere inexplicable vagaries
  of madness ; for I beheld him gazing upon vacancy for long hours , in an attitude of the profoundest attention , as if listening to some imaginary sound . It was no wonder that his condition terrified\u2014that it infected me . I felt creeping upon me , by slow yet certain degrees , the wild influence of his own fantastic yet impressive superstitions . It was , especially , upon retiring to bed late in the night of the seventh or eighth day after the placing of the lady Madeline within the donjon , that I experienced the full power of such feelings . Sleep came not near my couch , while the hours waned and waned away . I struggled to reason off the nervousness which had dominion over me . I endeavored to believe that much , if not all of what I felt , was due to the bewildering influence of the gloomy furniture of the room\u2014of the dark and tattered draperies , which , tortured into motion by the breath of a rising tempest , swayed fitfully to and fro upon the walls , and rustled unea
 sily about the decorations of the bed . But my efforts were fruitless . An irrepressible tremor gradually pervaded my frame ; and , at length , there sat upon my very heart an incubus of utterly causeless alarm . Shaking this off with a gasp and a struggle , I uplifted myself upon the pillows , and peering earnestly within the intense darkness of the chamber , hearkened\u2014I know not why , except that an instinctive spirit prompted me\u2014to certain low and indefinite sounds which came , through the pauses of the storm , at long intervals , I knew not whence . Overpowered by an intense sentiment of horror , unaccountable yet unendurable , I threw on my clothes with haste ( for I felt that I should sleep no more during the night ) , and endeavored to arouse myself from the pitiable condition into which I had fallen , by pacing rapidly to and fro through the apartment . I had taken but few turns in this manner , when a light step on an adjoining staircase arrested my attention . I presen
 tly recognized it as that of Usher . In an instant afterward he rapped , with a gentle touch , at my door , and entered , bearing a lamp . His countenance was , as usual , cadaverously wan\u2014but , moreover , there was a species of mad hilarity in his eyes\u2014and evidently restrained hysteria in his whole demeanor . His air appalled me\u2014but anything was preferable to the solitude which I had so long endured , and I even welcomed his presence as a relief . " And you have not seen it ? " he said abruptly , after having stared about him for some moments in silence\u2014"You have not then seen it ? \u2014 but stay ! you shall . " Thus speaking , and having carefully shaded his lamp , he hurried to one of the casements , and threw it freely open to the storm . The impetuous fury of the entering gust nearly lifted us from our feet . It was , indeed , a tempestuous yet sternly beautiful night , and one wildly singular in its terror and its beauty . A whirlwind had apparently collected its force i
 n our vicinity ; for there were frequent and violent alterations in the direction of the wind ; and the exceeding density of the clouds ( which hung so low as to press upon the turrets of the house ) did not prevent our perceiving the lifelike velocity with which they flew careering from all points against each other , without passing away into the distance . I say that even their exceeding density did not prevent our perceiving this\u2014yet we had no glimpse of the moon or stars\u2014nor was there any flashing forth of the lightning . But the under surfaces of the huge masses of agitated vapor , as well as all terrestrial objects immediately around us , were glowing in the unnatural light of a faintly luminous and distinctly visible gaseous exhalation which hung about and enshrouded the mansion . " You must not\u2014you shall not behold this ! " said I , shudderingly , to Usher , as I led him , with a gentle violence , from the window to a seat . " These appearances , which bewilder you , 
 are merely electrical phenomena not uncommon\u2014or it may be that they have their ghastly origin in the rank miasma of the tarn . Let us close this casement\u2014the air is chilling and dangerous to your frame . Here is one of your favorite romances . I will read and you shall listen ; \u2014 and so we will pass away this terrible night together . " The antique volume which I had taken up was the " Mad Trist " of Sir Launcelot Canning ; but I had called it a favorite of Usher 's more in sad jest than in earnest ; for , in truth , there is little in its uncouth and unimaginative prolixity which could have had interest for the lofty and spiritual ideality of my friend . It was , however , the only book immediately at hand ; and I indulged a vague hope that the excitement which now agitated the hypochondriac , might find relief ( for the history of mental disorder is full of similar anomalies ) even in the extremeness of the folly which I should read . Could I have judged , indeed , by the wi
 ld , overstrained air of vivacity with which he hearkened , or apparently harkened , to the words of the tale , I might well have congratulated myself upon the success of my design . I had arrived at that well-known portion of the story where Ethelred , the hero of the Trist , having sought in vain for peaceable admission into the dwelling of the hermit , proceeds to make good an entrance by force . Here , it will be remembered , the words of the narrative run thus : " And Ethelred , who was by nature of a doughty heart , and who was now mighty withal , on account of the powerfulness of the wine which he had drunken , waited no longer to hold parley with the hermit , who , in sooth , was of an obstinate and maliceful turn ; but , feeling the rain upon his shoulders , and fearing the rising of the tempest , uplifted his mace outright , and , with blows , made quickly room in the plankings of the door for his gauntleted hand ; and now pulling therewith sturdily , he so cracked , and r
 ipped , and tore all asunder , that the noise of the dry and hollow-sounding wood alarummed and reverberated throughout the forest . " At the termination of this sentence I started , and for a moment paused ; for it appeared to me ( although I at once concluded that my excited fancy had deceived me)\u2014it appeared to me that , from some very remote portion of the mansion , there came , indistinctly , to my ears what might have been , in its exact similarity of character , the echo ( but a stifled and dull one certainly ) of the very cracking and ripping sound which Sir Launcelot had so particularly described . It was , beyond doubt , the coincidence alone which had arrested my attention ; for , amid the rattling of the sashes of the casements , and the ordinary commingled noises of the still increasing storm , the sound , in itself , had nothing , surely , which should have interested or disturbed me . I continued the story : " But the good champion Ethelred , now entering within the
  door , was sore enraged and amazed to perceive no signal of the maliceful hermit ; but , in the stead thereof , a dragon of a scaly and prodigious demeanor , and of a fiery tongue , which sate in guard before a palace of gold , with a floor of silver ; and upon the wall there hung a shield of shining brass with this legend enwritten \u2014 Who entereth herein , a conqueror hath bin ; Who slayeth the dragon , the shield he shall win ; And Ethelred uplifted his mace , and struck upon the head of the dragon , which fell before him , and gave up his pesty breath , with a shriek so horrid and harsh , and withal so piercing , that Ethelred had fain to close his ears with his hands against the dreadful noise of it , the like whereof was never before heard . " Here again I paused abruptly , and now with a feeling of wild amazement\u2014for there could be no doubt whatever that , in this instance , I did actually hear ( although from what direction it proceeded I found it impossible to say ) a lo
 w and apparently distant , but harsh , protracted , and most unusual screaming or grating sound\u2014the exact counterpart of what my fancy had already conjured up for the dragon 's unnatural shriek as described by the romancer . Oppressed , as I certainly was , upon the occurrence of this second and most extraordinary coincidence , by a thousand conflicting sensations , in which wonder and extreme terror were predominant , I still retained sufficient presence of mind to avoid exciting , by any observation , the sensitive nervousness of my companion . I was by no means certain that he had noticed the sounds in question ; although , assuredly , a strange alteration had , during the last few minutes , taken place in his demeanor . From a position fronting my own , he had gradually brought round his chair , so as to sit with his face to the door of the chamber ; and thus I could but partially perceive his features , although I saw that his lips trembled as if he were murmuring inaudibly .
  His head had dropped upon his breast\u2014yet I knew that he was not asleep , from the wide and rigid opening of the eye as I caught a glance of it in profile . The motion of his body , too , was at variance with this idea\u2014for he rocked from side to side with a gentle yet constant and uniform sway . Having rapidly taken notice of all this , I resumed the narrative of Sir Launcelot , which thus proceeded : " And now the champion , having escaped from the terrible fury of the dragon , bethinking himself of the brazen shield , and of the breaking up of the enchantment which was upon it , removed the carcass from out of the way before him , and approached valorously over the silver pavement of the castle to where the shield was upon the wall ; which in sooth tarried not for his full coming , but fell down at his feet upon the silver floor , with a mighty great and terrible ringing sound . " No sooner had these syllables passed my lips , than\u2014as if a shield of brass had indeed , at the
  moment , fallen heavily upon a floor of silver\u2014I became aware of a distinct , hollow , metallic and clangorous , yet apparently muffled reverberation . Completely unnerved , I leaped to my feet ; but the measured rocking movement of Usher was undisturbed . I rushed to the chair in which he sat . His eyes were bent fixedly before him , and throughout his whole countenance there reigned a stony rigidity . But , as I placed my hand upon his shoulder , there came a strong shudder over his whole person ; a sickly smile quivered about his lips ; and I saw that he spoke in a low , hurried , and gibbering murmur , as if unconscious of my presence . Bending closely over him , I at length drank in the hideous import of his words . " Not hear it ? \u2014 yes , I hear it , and have heard it . Long\u2014long\u2014long\u2014many minutes , many hours , many days , have I heard it\u2014yet I dared not\u2014oh , pity me , miserable wretch that I am ! \u2014 I dared not\u2014I dared not speak ! We have put her living in 
 the tomb ! Said I not that my senses were acute ? I now tell you that I heard her first feeble movements in the hollow coffin . I heard them\u2014many , many days ago\u2014yet I dared not\u2014I dared not speak ! And now\u2014to-night\u2014Ethelred\u2014ha ! ha ! \u2014 the breaking of the hermit 's door , and the death-cry of the dragon , and the clangor of the shield ! \u2014 say , rather , the rending of her coffin , and the grating of the iron hinges of her prison , and her struggles within the coppered archway of the vault ! Oh , whither shall I fly ? Will she not be here anon ? Is she not hurrying to upbraid me for my haste ? Have I not heard her footstep on the stair ? Do I not distinguish that heavy and horrible beating of her heart ? Madman ! " \u2014 here he sprang furiously to his feet , and shrieked out his syllables , as if in the effort he were giving up his soul\u2014"Madman ! I tell you that she now stands without the door ! " As if in the superhuman energy of his utterance there had been found t
 he potency of a spell\u2014the huge antique pannels to which the speaker pointed threw slowly back , upon the instant , their ponderous and ebony jaws . It was the work of the rushing gust\u2014but then without those doors there did stand the lofty and enshrouded figure of the lady Madeline of Usher . There was blood upon her white robes , and the evidence of some bitter struggle upon every portion of her emaciated frame . For a moment she remained trembling and reeling to and fro upon the threshold\u2014then , with a low , moaning cry , fell heavily inward upon the person of her brother , and in her violent and now final death-agonies , bore him to the floor a corpse , and a victim to the terrors he had anticipated . From that chamber , and from that mansion , I fled aghast . The storm was still abroad in all its wrath as I found myself crossing the old causeway . Suddenly there shot along the path a wild light , and I turned to see whence a gleam so unusual could have issued ; for the vast
  house and its shadows were alone behind me . The radiance was that of the full , setting , and blood-red moon , which now shone vividly through that once barely discernible fissure , of which I have before spoken as extending from the roof of the building , in a zigzag direction , to the base . While I gazed , this fissure rapidly widened\u2014there came a fierce breath of the whirlwind\u2014the entire orb of the satellite burst at once upon my sight\u2014my brain reeled as I saw the mighty walls rushing asunder\u2014there was a long tumultuous shouting sound like the voice of a thousand waters\u2014and the deep and dank tarn at my feet closed sullenly and silently over the fragments of the " House of Usher . " 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/66FictTolstoyL_AnnaK_I10_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/66FictTolstoyL_AnnaK_I10_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/66FictTolstoyL_AnnaK_I10_EN.txt.txt
new file mode 100644
index 0000000..7680d55
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/66FictTolstoyL_AnnaK_I10_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Part One , Chapter 10 When Levin went into the restaurant with Oblonsky , he could not help noticing a certain peculiarity of expression , as it were , a restrained radiance , about the face and whole figure of Stepan Arkadyevitch . Oblonsky took off his overcoat , and with his hat over one ear walked into the dining room , giving directions to the Tatar waiters , who were clustered about him in evening coats , bearing napkins . Bowing to right and left to the people he met , and here as everywhere joyously greeting acquaintances , he went up to the sideboard for a preliminary appetizer of fish and vodka , and said to the painted Frenchwoman decked in ribbons , lace , and ringlets , behind the counter , something so amusing that even that Frenchwoman was moved to genuine laughter . Levin for his part refrained from taking any vodka simply because he felt such a loathing of that Frenchwoman , all made up , it seemed , of false hair , poudre de riz , and vinaigre de toilette . He 
 made haste to move away from her , as from a dirty place . His whole soul was filled with memories of Kitty , and there was a smile of triumph and happiness shining in his eyes . " This way , your excellency , please . Your excellency wo n't be disturbed here , " said a particularly pertinacious , white-headed old Tatar with immense hips and coat-tails gaping widely behind . " Walk in , your excellency , " he said to Levin ; by way of showing his respect to Stepan Arkadyevitch , being attentive to his guest as well . Instantly flinging a fresh cloth over the round table under the bronze chandelier , though it already had a table cloth on it , he pushed up velvet chairs , and came to a standstill before Stepan Arkadyevitch with a napkin and a bill of fare in his hands , awaiting his commands . " If you prefer it , your excellency , a private room will be free directly ; Prince Golistin with a lady . Fresh oysters have come in . " " Ah ! oysters . " Stepan Arkadyevitch became thoughtf
 ul . " How if we were to change our program , Levin ? " he said , keeping his finger on the bill of fare . And his face expressed serious hesitation . " Are the oysters good ? Mind now . " " They 're Flensburg , your excellency . We 've no Ostend . " " Flensburg will do , but are they fresh ? " " Only arrived yesterday . " " Well , then , how if we were to begin with oysters , and so change the whole program ? Eh ? " " It 's all the same to me . I should like cabbage soup and porridge better than anything ; but of course there 's nothing like that here . " " Porridge � la Russe , your honor would like ? " said the Tatar , bending down to Levin , like a nurse speaking to a child . " No , joking apart , whatever you choose is sure to be good . I 've been skating , and I 'm hungry . And do n't imagine , " he added , detecting a look of dissatisfaction on Oblonsky 's face , " that I sha n't appreciate your choice . I am fond of good things . " " I should hope so ! After all , it 's one
  of the pleasures of life , " said Stepan Arkadyevitch . " Well , then , my friend , you give us two\u2014or better say three\u2014dozen oysters , clear soup with vegetables ... " " Printaniere , " prompted the Tatar . But Stepan Arkadyevitch apparently did not care to allow him the satisfaction of giving the French names of the dishes . " With vegetables in it , you know . Then turbot with thick sauce , then ... roast beef ; and mind it 's good . Yes , and capons , perhaps , and then sweets . " The Tatar , recollecting that it was Stepan Arkadyevitch 's way not to call the dishes by the names in the French bill of fare , did not repeat them after him , but could not resist rehearsing the whole menu to himself according to the bill : \u2014 " Soupe printani�re , turbot , sauce Beaumarchais , poulard � l'estragon , mac�doine de fruits ... etc. , " and then instantly , as though worked by springs , laying down one bound bill of fare , he took up another , the list of wines , and submitted i
 t to Stepan Arkadyevitch . " What shall we drink ? " " What you like , only not too much . Champagne , " said Levin . " What ! to start with ? You 're right though , I dare say . Do you like the white seal ? " " Cachet blanc , " prompted the Tatar . " Very well , then , give us that brand with the oysters , and then we 'll see . " " Yes , sir . And what table wine ? " " You can give us Nuits . Oh , no , better the classic Chablis . " " Yes , sir . And your cheese , your excellency ? " " Oh , yes , Parmesan . Or would you like another ? " " No , it 's all the same to me , " said Levin , unable to suppress a smile . And the Tatar ran off with flying coat-tails , and in five minutes darted in with a dish of opened oysters on mother-of-pearl shells , and a bottle between his fingers . Stepan Arkadyevitch crushed the starchy napkin , tucked it into his waistcoat , and settling his arms comfortably , started on the oysters . " Not bad , " he said , stripping the oysters from the pearly sh
 ell with a silver fork , and swallowing them one after another . " Not bad , " he repeated , turning his dewy , brilliant eyes from Levin to the Tatar . Levin ate the oysters indeed , though white bread and cheese would have pleased him better . But he was admiring Oblonsky . Even the Tatar , uncorking the bottle and pouring the sparkling wine into the delicate glasses , glanced at Stepan Arkadyevitch , and settled his white cravat with a perceptible smile of satisfaction . " You do n't care much for oysters , do you ? " said Stepan Arkadyevitch , emptying his wine glass , " or you 're worried about something . Eh ? " He wanted Levin to be in good spirits . But it was not that Levin was not in good spirits ; he was ill at ease . With what he had in his soul , he felt sore and uncomfortable in the restaurant , in the midst of private rooms where men were dining with ladies , in all this fuss and bustle ; the surroundings of bronzes , looking glasses , gas , and waiters\u2014all of it wa
 s offensive to him . He was afraid of sullying what his soul was brimful of . " I ? Yes , I am ; but besides , all this bothers me , " he said . " You ca n't conceive how queer it all seems to a country person like me , as queer as that gentleman 's nails I saw at your place ... " " Yes , I saw how much interested you were in poor Grinevitch 's nails , " said Stepan Arkadyevitch , laughing . " It 's too much for me , " responded Levin . " Do try , now , and put yourself in my place , take the point of view of a country person . We in the country try to bring our hands into such a state as will be most convenient for working with . So we cut our nails ; sometimes we turn up our sleeves . And here people purposely let their nails grow as long as they will , and link on small saucers by way of studs , so that they can do nothing with their hands . " Stepan Arkadyevitch smiled gaily . " Oh , yes , that 's just a sign that he has no need to do coarse work . His work is with the mind ... 
 " " Maybe . But still it 's queer to me , just as at this moment it seems queer to me that we country folks try to get our meals over as soon as we can , so as to be ready for our work , while here are we trying to drag out our meal as long as possible , and with that object eating oysters ... " " Why , of course , " objected Stepan Arkadyevitch . " But that 's just the aim of civilization\u2014to make everything a source of enjoyment . " " Well , if that 's its aim , I 'd rather be a savage . " " And so you are a savage . All you Levins are savages . " Levin sighed . He remembered his brother Nikolay , and felt ashamed and sore , and he scowled ; but Oblonsky began speaking of a subject which at once drew his attention . " Oh , I say , are you going tonight to our people , the Shtcherbatskys ' , I mean ? " he said , his eyes sparkling significantly as he pushed away the empty rough shells , and drew the cheese towards him . " Yes , I shall certainly go , " replied Levin ; " though I f
 ancied the princess was not very warm in her invitation . " " What nonsense ! That 's her manner ... . Come , boy , the soup ! ... . That 's her manner\u2014grande dame , " said Stepan Arkadyevitch . " I 'm coming , too , but I have to go to the Countess Bonina 's rehearsal . Come , is n't it true that you 're a savage ? How do you explain the sudden way in which you vanished from Moscow ? The Shtcherbatskys were continually asking me about you , as though I ought to know . The only thing I know is that you always do what no one else does . " " Yes , " said Levin , slowly and with emotion , " you 're right . I am a savage . Only , my savageness is not in having gone away , but in coming now . Now I have come ... " " Oh , what a lucky fellow you are ! " broke in Stepan Arkadyevitch , looking into Levin 's eyes . " Why ? " " I know a gallant steed by tokens sure , And by his eyes I know a youth in love , " declaimed Stepan Arkadyevitch . " Everything is before you . " " Why , is it over 
 for you already ? " " No ; not over exactly , but the future is yours , and the present is mine , and the present\u2014well , it 's not all that it might be . " " How so ? " " Oh , things go wrong . But I do n't want to talk of myself , and besides I ca n't explain it all , " said Stepan Arkadyevitch . " Well , why have you come to Moscow , then ? ... . Hi ! take away ! " he called to the Tatar . " You guess ? " responded Levin , his eyes like deep wells of light fixed on Stepan Arkadyevitch . " I guess , but I ca n't be the first to talk about it . You can see by that whether I guess right or wrong , " said Stepan Arkadyevitch , gazing at Levin with a subtle smile . " Well , and what have you to say to me ? " said Levin in a quivering voice , feeling that all the muscles of his face were quivering too . " How do you look at the question ? " Stepan Arkadyevitch slowly emptied his glass of Chablis , never taking his eyes off Levin . " I ? " said Stepan Arkadyevitch , " there 's nothing 
 I desire so much as that\u2014nothing ! It would be the best thing that could be . " " But you 're not making a mistake ? You know what we 're speaking of ? " said Levin , piercing him with his eyes . " You think it 's possible ? " " I think it 's possible . Why not possible ? " " No ! do you really think it 's possible ? No , tell me all you think ! Oh , but if ... if refusal 's in store for me ! ... Indeed I feel sure ... " " Why should you think that ? " said Stepan Arkadyevitch , smiling at his excitement . " It seems so to me sometimes . That will be awful for me , and for her too . " " Oh , well , anyway there 's nothing awful in it for a girl . Every girl 's proud of an offer . " " Yes , every girl , but not she . " Stepan Arkadyevitch smiled . He so well knew that feeling of Levin 's , that for him all the girls in the world were divided into two classes : one class\u2014all the girls in the world except her , and those girls with all sorts of human weaknesses , and very ordinary 
 girls : the other class\u2014she alone , having no weaknesses of any sort and higher than all humanity . " Stay , take some sauce , " he said , holding back Levin 's hand as it pushed away the sauce . Levin obediently helped himself to sauce , but would not let Stepan Arkadyevitch go on with his dinner . " No , stop a minute , stop a minute , " he said . " You must understand that it 's a question of life and death for me . I have never spoken to any one of this . And there 's no one I could speak of it to , except you . You know we 're utterly unlike each other , different tastes and views and everything ; but I know you 're fond of me and understand me , and that 's why I like you awfully . But for God 's sake , be quite straightforward with me . " " I tell you what I think , " said Stepan Arkadyevitch , smiling . " But I 'll say more : my wife is a wonderful woman ... " Stepan Arkadyevitch sighed , remembering his position with his wife , and , after a moment 's silence , resumed\u2014
 "She has a gift of foreseeing things . She sees right through people ; but that 's not all ; she knows what will come to pass , especially in the way of marriages . She foretold , for instance , that Princess Shahovskaya would marry Brenteln . No one would believe it , but it came to pass . And she 's on your side . " " How do you mean ? " " It 's not only that she likes you\u2014she says that Kitty is certain to be your wife . " At these words Levin 's face suddenly lighted up with a smile , a smile not far from tears of emotion . " She says that ! " cried Levin . " I always said she was exquisite , your wife . There , that 's enough , enough said about it , " he said , getting up from his seat . " All right , but do sit down . " But Levin could not sit down . He walked with his firm tread twice up and down the little cage of a room , blinked his eyelids that his tears might not fall , and only then sat down to the table . " You must understand , " said he , " it 's not love . I 've b
 een in love , but it 's not that . It 's not my feeling , but a sort of force outside me has taken possession of me . I went away , you see , because I made up my mind that it could never be , you understand , as a happiness that does not come on earth ; but I 've struggled with myself , I see there 's no living without it . And it must be settled . " " What did you go away for ? " " Ah , stop a minute ! Ah , the thoughts that come crowding on one ! The questions one must ask oneself ! Listen . You ca n't imagine what you 've done for me by what you said . I 'm so happy that I 've become positively hateful ; I 've forgotten everything . I heard today that my brother Nikolay ... you know , he 's here ... I had even forgotten him . It seems to me that he 's happy too . It 's a sort of madness . But one thing 's awful ... . Here , you 've been married , you know the feeling ... it 's awful that we\u2014old\u2014with a past ... not of love , but of sins ... are brought all at once so near to 
 a creature pure and innocent ; it 's loathsome , and that 's why one ca n't help feeling oneself unworthy . " " Oh , well , you 've not many sins on your conscience . " " Alas ! all the same , " said Levin , " when with loathing I go over my life , I shudder and curse and bitterly regret it ... . Yes . " " What would you have ? The world 's made so , " said Stepan Arkadyevitch . " The one comfort is like that prayer , which I always liked : 'Forgive me not according to my unworthiness , but according to Thy lovingkindness . ' That 's the only way she can forgive me . " 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/67FictTolstoyL_AnnaK_I11_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/67FictTolstoyL_AnnaK_I11_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/67FictTolstoyL_AnnaK_I11_EN.txt.txt
new file mode 100644
index 0000000..b5a6f30
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/67FictTolstoyL_AnnaK_I11_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Part One , Chapter 11 Levin emptied his glass , and they were silent for a while . " There 's one other thing I ought to tell you . Do you know Vronsky ? " Stepan Arkadyevitch asked Levin . " No , I do n't . Why do you ask ? " " Give us another bottle , " Stepan Arkadyevitch directed the Tatar , who was filling up their glasses and fidgeting round them just when he was not wanted . " Why you ought to know Vronsky is that he 's one of your rivals . " " Who 's Vronsky ? " said Levin , and his face was suddenly transformed from the look of childlike ecstasy which Oblonsky had just been admiring to an angry and unpleasant expression . " Vronsky is one of the sons of Count Kirill Ivanovitch Vronsky , and one of the finest specimens of the gilded youth of Petersburg . I made his acquaintance in Tver when I was there on official business , and he came there for the levy of recruits . Fearfully rich , handsome , great connections , an aide-de-camp , and with all that a very nice , good-
 natured fellow . But he 's more than simply a good-natured fellow , as I 've found out here\u2014he 's a cultivated man , too , and very intelligent ; he 's a man who 'll make his mark . " Levin scowled and was dumb . " Well , he turned up here soon after you 'd gone , and as I can see , he 's over head and ears in love with Kitty , and you know that her mother ... " " Excuse me , but I know nothing , " said Levin , frowning gloomily . And immediately he recollected his brother Nikolay and how hateful he was to have been able to forget him . " You wait a bit , wait a bit , " said Stepan Arkadyevitch , smiling and touching his hand . " I 've told you what I know , and I repeat that in this delicate and tender matter , as far as one can conjecture , I believe the chances are in your favor . " Levin dropped back in his chair ; his face was pale . " But I would advise you to settle the thing as soon as may be , " pursued Oblonsky , filling up his glass . " No , thanks , I ca n't drink any 
 more , " said Levin , pushing away his glass . " I shall be drunk ... . Come , tell me how are you getting on ? " he went on , obviously anxious to change the conversation . " One word more : in any case I advise you to settle the question soon . Tonight I do n't advise you to speak , " said Stepan Arkadyevitch . " Go round tomorrow morning , make an offer in due form , and God bless you ... " " Oh , do you still think of coming to me for some shooting ? Come next spring , do , " said Levin . Now his whole soul was full of remorse that he had begun this conversation with Stepan Arkadyevitch . A feeling such as his was profaned by talk of the rivalry of some Petersburg officer , of the suppositions and the counsels of Stepan Arkadyevitch . Stepan Arkadyevitch smiled . He knew what was passing in Levin 's soul . " I 'll come some day , " he said . " But women , my boy , they 're the pivot everything turns upon . Things are in a bad way with me , very bad . And it 's all through women 
 . Tell me frankly now , " he pursued , picking up a cigar and keeping one hand on his glass ; " give me your advice . " " Why , what is it ? " " I 'll tell you . Suppose you 're married , you love your wife , but you 're fascinated by another woman ... " " Excuse me , but I 'm absolutely unable to comprehend how ... just as I ca n't comprehend how I could now , after my dinner , go straight to a baker 's shop and steal a roll . " Stepan Arkadyevitch 's eyes sparkled more than usual . " Why not ? A roll will sometimes smell so good one ca n't resist it . " " Himmlisch ist 's , wenn ich bezwungen Meine irdische Begier ; Aber doch wenn 's nich gelungen Hatt ' ich auch recht huebsch Plaisir ! " As he said this , Stepan Arkadyevitch smiled subtly . Levin , too , could not help smiling . " Yes , but joking apart , " resumed Stepan Arkadyevitch , " you must understand that the woman is a sweet , gentle loving creature , poor and lonely , and has sacrificed everything . Now , when the thing
  's done , do n't you see , can one possibly cast her off ? Even supposing one parts from her , so as not to break up one 's family life , still , can one help feeling for her , setting her on her feet , softening her lot ? " " Well , you must excuse me there . You know to me all women are divided into two classes ... at least no ... truer to say : there are women and there are ... I 've never seen exquisite fallen beings , and I never shall see them , but such creatures as that painted Frenchwoman at the counter with the ringlets are vermin to my mind , and all fallen women are the same . " " But the Magdalen ? " " Ah , drop that ! Christ would never have said those words if He had known how they would be abused . Of all the Gospel those words are the only ones remembered . However , I 'm not saying so much what I think , as what I feel . I have a loathing for fallen women . You 're afraid of spiders , and I of these vermin . Most likely you 've not made a study of spiders and do n
 't know their character ; and so it is with me . " " It 's very well for you to talk like that ; it 's very much like that gentleman in Dickens who used to fling all difficult questions over his right shoulder . But to deny the facts is no answer . What 's to be done\u2014you tell me that , what 's to be done ? Your wife gets older , while you 're full of life . Before you 've time to look round , you feel that you ca n't love your wife with love , however much you may esteem her . And then all at once love turns up , and you 're done for , done for , " Stepan Arkadyevitch said with weary despair . Levin half smiled . " Yes , you 're done for , " resumed Oblonsky . " But what 's to be done ? " " Do n't steal rolls . " Stepan Arkadyevitch laughed outright . " Oh , moralist ! But you must understand , there are two women ; one insists only on her rights , and those rights are your love , which you ca n't give her ; and the other sacrifices everything for you and asks for nothing . What a
 re you to do ? How are you to act ? There 's a fearful tragedy in it . " " If you care for my profession of faith as regards that , I 'll tell you that I do n't believe there was any tragedy about it . And this is why . To my mind , love ... both the sorts of love , which you remember Plato defines in his Banquet , served as the test of men . Some men only understand one sort , and some only the other . And those who only know the non-platonic love have no need to talk of tragedy . In such love there can be no sort of tragedy . 'I 'm much obliged for the gratification , my humble respects'\u2014that 's all the tragedy . And in platonic love there can be no tragedy , because in that love all is clear and pure , because ... " At that instant Levin recollected his own sins and the inner conflict he had lived through . And he added unexpectedly : " But perhaps you are right . Very likely ... I do n't know , I do n't know . " " It 's this , do n't you see , " said Stepan Arkadyevitch , " yo
 u 're very much all of a piece . That 's your strong point and your failing . You have a character that 's all of a piece , and you want the whole of life to be of a piece too\u2014but that 's not how it is . You despise public official work because you want the reality to be invariably corresponding all the while with the aim\u2014and that 's not how it is . You want a man 's work , too , always to have a defined aim , and love and family life always to be undivided\u2014and that 's not how it is . All the variety , all the charm , all the beauty of life is made up of light and shadow . " Levin sighed and made no reply . He was thinking of his own affairs , and did not hear Oblonsky . And suddenly both of them felt that though they were friends , though they had been dining and drinking together , which should have drawn them closer , yet each was thinking only of his own affairs , and they had nothing to do with one another . Oblonsky had more than once experienced this extreme sense of al
 oofness , instead of intimacy , coming on after dinner , and he knew what to do in such cases . " Bill ! " he called , and he went into the next room where he promptly came across an aide-de-camp of his acquaintance and dropped into conversation with him about an actress and her protector . And at once in the conversation with the aide-de-camp Oblonsky had a sense of relaxation and relief after the conversation with Levin , which always put him to too great a mental and spiritual strain . When the Tatar appeared with a bill for twenty-six roubles and odd kopecks , besides a tip for himself , Levin , who would another time have been horrified , like any one from the country , at his share of fourteen roubles , did not notice it , paid , and set off homewards to dress and go to the Shtcherbatskys ' there to decide his fate . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/68FictTolstoyL_WarPeace_I16_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/68FictTolstoyL_WarPeace_I16_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/68FictTolstoyL_WarPeace_I16_EN.txt.txt
new file mode 100644
index 0000000..4f57251
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/68FictTolstoyL_WarPeace_I16_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff CHAPTER XIX At the men 's end of the table the talk grew more and more animated . The colonel told them that the declaration of war had already appeared in Petersburg and that a copy , which he had himself seen , had that day been forwarded by courier to the commander in chief . " And why the deuce are we going to fight Bonaparte ? " remarked Shinshin . " He has stopped Austria 's cackle and I fear it will be our turn next . " The colonel was a stout , tall , plethoric German , evidently devoted to the service and patriotically Russian . He resented Shinshin 's remark . " It is for the reasson , my goot sir , " said he , speaking with a German accent , " for the reasson zat ze Emperor knows zat . He declares in ze manifessto zat he cannot fiew wiz indifference ze danger vreatening Russia and zat ze safety and dignity of ze Empire as vell as ze sanctity of its alliances ... " he spoke this last word with particular emphasis as if in it lay the gist of the matter . Then with the u
 nerring official memory that characterized him he repeated from the opening words of the manifesto : ... and the wish , which constitutes the Emperor 's sole and absolute aim--to establish peace in Europe on firm foundations--has now decided him to despatch part of the army abroad and to create a new condition for the attainment of that purpose . " Zat , my dear sir , is vy ... " he concluded , drinking a tumbler of wine with dignity and looking to the count for approval . " Connaissez-vous le Proverbe : * 'Jerome , Jerome , do not roam , but turn spindles at home ! ' ? " said Shinshin , puckering his brows and smiling . " Cela nous convient a merveille . *[2 ] Suvorov now--he knew what he was about ; yet they beat him a plate couture , *[3 ] and where are we to find Suvorovs now ? Je vous demande un peu , " *[4 ] said he , continually changing from French to Russian . * Do you know the proverb ? [ 2 ] That suits us down to the ground . [ 3 ] Hollow . [ 4 ] I just ask you that . " V
 e must vight to the last tr-r-op of our plood ! " said the colonel , thumping the table ; " and ve must tie for our Emperor , and zen all vill pe vell . And ve must discuss it as little as po-o-ossible " ... he dwelt particularly on the word possible ... " as po-o-ossible , " he ended , again turning to the count . " Zat is how ve old hussars look at it , and zere 's an end of it ! And how do you , a young man and a young hussar , how do you judge of it ? " he added , addressing Nicholas , who when he heard that the war was being discussed had turned from his partner with eyes and ears intent on the colonel . " I am quite of your opinion , " replied Nicholas , flaming up , turning his plate round and moving his wineglasses about with as much decision and desperation as though he were at that moment facing some great danger . " I am convinced that we Russians must die or conquer , " he concluded , conscious--as were others--after the words were uttered that his remarks were too enthu
 siastic and emphatic for the occasion and were therefore awkward . " What you said just now was splendid ! " said his partner Julie . Sonya trembled all over and blushed to her ears and behind them and down to her neck and shoulders while Nicholas was speaking . Pierre listened to the colonel 's speech and nodded approvingly . " That 's fine , " said he . " The young man 's a real hussar ! " shouted the colonel , again thumping the table . " What are you making such a noise about over there ? " Marya Dmitrievna 's deep voice suddenly inquired from the other end of the table . " What are you thumping the table for ? " she demanded of the hussar , " and why are you exciting yourself ? Do you think the French are here ? " " I am speaking ze truce , " replied the hussar with a smile . " It 's all about the war , " the count shouted down the table . " You know my son 's going , Marya Dmitrievna ? My son is going . " " I have four sons in the army but still I do n't fret . It is all in Go
 d 's hands . You may die in your bed or God may spare you in a battle , " replied Marya Dmitrievna 's deep voice , which easily carried the whole length of the table . " That 's true ! " Once more the conversations concentrated , the ladies ' at the one end and the men 's at the other . " You wo n't ask , " Natasha 's little brother was saying ; " I know you wo n't ask ! " " I will , " replied Natasha . Her face suddenly flushed with reckless and joyous resolution . She half rose , by a glance inviting Pierre , who sat opposite , to listen to what was coming , and turning to her mother : " Mamma ! " rang out the clear contralto notes of her childish voice , audible the whole length of the table . " What is it ? " asked the countess , startled ; but seeing by her daughter 's face that it was only mischief , she shook a finger at her sternly with a threatening and forbidding movement of her head . The conversation was hushed . " Mamma ! What sweets are we going to have ? " and Natasha
  's voice sounded still more firm and resolute . The countess tried to frown , but could not . Marya Dmitrievna shook her fat finger . " Cossack ! " she said threateningly . Most of the guests , uncertain how to regard this sally , looked at the elders . " You had better take care ! " said the countess . " Mamma ! What sweets are we going to have ? " Natasha again cried boldly , with saucy gaiety , confident that her prank would be taken in good part . Sonya and fat little Petya doubled up with laughter . " You see ! I have asked , " whispered Natasha to her little brother and to Pierre , glancing at him again . " Ice pudding , but you wo n't get any , " said Marya Dmitrievna . Natasha saw there was nothing to be afraid of and so she braved even Marya Dmitrievna . " Marya Dmitrievna ! What kind of ice pudding ? I do n't like ice cream . " " Carrot ices . " " No ! What kind , Marya Dmitrievna ? What kind ? " she almost screamed ; " I want to know ! " Marya Dmitrievna and the countess
  burst out laughing , and all the guests joined in . Everyone laughed , not at Marya Dmitrievna 's answer but at the incredible boldness and smartness of this little girl who had dared to treat Marya Dmitrievna in this fashion . Natasha only desisted when she had been told that there would be pineapple ice . Before the ices , champagne was served round . The band again struck up , the count and countess kissed , and the guests , leaving their seats , went up to " congratulate " the countess , and reached across the table to clink glasses with the count , with the children , and with one another . Again the footmen rushed about , chairs scraped , and in the same order in which they had entered but with redder faces , the guests returned to the drawing room and to the count 's study . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/69FictTolstoyL_WarPeace_I2_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/69FictTolstoyL_WarPeace_I2_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/69FictTolstoyL_WarPeace_I2_EN.txt.txt
new file mode 100644
index 0000000..698a1ea
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/69FictTolstoyL_WarPeace_I2_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff CHAPTER II Anna Pavlovna 's drawing room was gradually filling . The highest Petersburg society was assembled there : people differing widely in age and character but alike in the social circle to which they belonged . Prince Vasili 's daughter , the beautiful Helene , came to take her father to the ambassador 's entertainment ; she wore a ball dress and her badge as maid of honor . The youthful little Princess Bolkonskaya , known as la femme la plus seduisante de Petersbourg , * was also there . She had been married during the previous winter , and being pregnant did not go to any large gatherings , but only to small receptions . Prince Vasili 's son , Hippolyte , had come with Mortemart , whom he introduced . The Abbe Morio and many others had also come . * The most fascinating woman in Petersburg . To each new arrival Anna Pavlovna said , " You have not yet seen my aunt , " or " You do not know my aunt ? " and very gravely conducted him or her to a little old lady , wearing l
 arge bows of ribbon in her cap , who had come sailing in from another room as soon as the guests began to arrive ; and slowly turning her eyes from the visitor to her aunt , Anna Pavlovna mentioned each one 's name and then left them . Each visitor performed the ceremony of greeting this old aunt whom not one of them knew , not one of them wanted to know , and not one of them cared about ; Anna Pavlovna observed these greetings with mournful and solemn interest and silent approval . The aunt spoke to each of them in the same words , about their health and her own , and the health of Her Majesty , " who , thank God , was better today . " And each visitor , though politeness prevented his showing impatience , left the old woman with a sense of relief at having performed a vexatious duty and did not return to her the whole evening . The young Princess Bolkonskaya had brought some work in a gold-embroidered velvet bag . Her pretty little upper lip , on which a delicate dark down was jus
 t perceptible , was too short for her teeth , but it lifted all the more sweetly , and was especially charming when she occasionally drew it down to meet the lower lip . As is always the case with a thoroughly attractive woman , her defect--the shortness of her upper lip and her half-open mouth--seemed to be her own special and peculiar form of beauty . Everyone brightened at the sight of this pretty young woman , so soon to become a mother , so full of life and health , and carrying her burden so lightly . Old men and dull dispirited young ones who looked at her , after being in her company and talking to her a little while , felt as if they too were becoming , like her , full of life and health . All who talked to her , and at each word saw her bright smile and the constant gleam of her white teeth , thought that they were in a specially amiable mood that day . The little princess went round the table with quick , short , swaying steps , her workbag on her arm , and gaily spreadin
 g out her dress sat down on a sofa near the silver samovar , as if all she was doing was a pleasure to herself and to all around her . " I have brought my work , " said she in French , displaying her bag and addressing all present . " Mind , Annette , I hope you have not played a wicked trick on me , " she added , turning to her hostess . " You wrote that it was to be quite a small reception , and just see how badly I am dressed . " And she spread out her arms to show her short-waisted , lace-trimmed , dainty gray dress , girdled with a broad ribbon just below the breast . " Soyez tranquille , Lise , you will always be prettier than anyone else , " replied Anna Pavlovna . " You know , " said the princess in the same tone of voice and still in French , turning to a general , " my husband is deserting me ? He is going to get himself killed . Tell me what this wretched war is for ? " she added , addressing Prince Vasili , and without waiting for an answer she turned to speak to his dau
 ghter , the beautiful Helene . " What a delightful woman this little princess is ! " said Prince Vasili to Anna Pavlovna . One of the next arrivals was a stout , heavily built young man with close-cropped hair , spectacles , the light-colored breeches fashionable at that time , a very high ruffle , and a brown dress coat . This stout young man was an illegitimate son of Count Bezukhov , a well-known grandee of Catherine 's time who now lay dying in Moscow . The young man had not yet entered either the military or civil service , as he had only just returned from abroad where he had been educated , and this was his first appearance in society . Anna Pavlovna greeted him with the nod she accorded to the lowest hierarchy in her drawing room . But in spite of this lowest-grade greeting , a look of anxiety and fear , as at the sight of something too large and unsuited to the place , came over her face when she saw Pierre enter . Though he was certainly rather bigger than the other men in
  the room , her anxiety could only have reference to the clever though shy , but observant and natural , expression which distinguished him from everyone else in that drawing room . " It is very good of you , Monsieur Pierre , to come and visit a poor invalid , " said Anna Pavlovna , exchanging an alarmed glance with her aunt as she conducted him to her . Pierre murmured something unintelligible , and continued to look round as if in search of something . On his way to the aunt he bowed to the little princess with a pleased smile , as to an intimate acquaintance . Anna Pavlovna 's alarm was justified , for Pierre turned away from the aunt without waiting to hear her speech about Her Majesty 's health . Anna Pavlovna in dismay detained him with the words : " Do you know the Abbe Morio ? He is a most interesting man . " " Yes , I have heard of his scheme for perpetual peace , and it is very interesting but hardly feasible . " " You think so ? " rejoined Anna Pavlovna in order to say s
 omething and get away to attend to her duties as hostess . But Pierre now committed a reverse act of impoliteness . First he had left a lady before she had finished speaking to him , and now he continued to speak to another who wished to get away . With his head bent , and his big feet spread apart , he began explaining his reasons for thinking the abbe 's plan chimerical . " We will talk of it later , " said Anna Pavlovna with a smile . And having got rid of this young man who did not know how to behave , she resumed her duties as hostess and continued to listen and watch , ready to help at any point where the conversation might happen to flag . As the foreman of a spinning mill , when he has set the hands to work , goes round and notices here a spindle that has stopped or there one that creaks or makes more noise than it should , and hastens to check the machine or set it in proper motion , so Anna Pavlovna moved about her drawing room , approaching now a silent , now a too-noisy 
 group , and by a word or slight rearrangement kept the conversational machine in steady , proper , and regular motion . But amid these cares her anxiety about Pierre was evident . She kept an anxious watch on him when he approached the group round Mortemart to listen to what was being said there , and again when he passed to another group whose center was the abbe . Pierre had been educated abroad , and this reception at Anna Pavlovna 's was the first he had attended in Russia . He knew that all the intellectual lights of Petersburg were gathered there and , like a child in a toyshop , did not know which way to look , afraid of missing any clever conversation that was to be heard . Seeing the self-confident and refined expression on the faces of those present he was always expecting to hear something very profound . At last he came up to Morio . Here the conversation seemed interesting and he stood waiting for an opportunity to express his own views , as young people are fond of doi
 ng . 
\ No newline at end of file


[25/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/80NewsGoalcom_MessiTop50_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/News/80NewsGoalcom_MessiTop50_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/80NewsGoalcom_MessiTop50_EN.txt.txt
new file mode 100644
index 0000000..579c641
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/80NewsGoalcom_MessiTop50_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Welcome to the Goal . com 50 ! In this special series , Goal . com editors worldwide vote for the top 50 players of 2010-11 . We count down to the announcement of the winner on August 22 with profiles of each and every player who made it into the top 50 ... Do n't Miss Goal . com 50 : Xavi ( 3 ) Goal . com 50 : Andres Iniesta ( 4 ) Goal . com 50 : Radamel Falcao ( 5 ) Goal . com 50 : Nemanja Vidic ( 6 ) On Wednesday night , Lionel Messi produced one of the finest individual performances seen on a football field in recent memory . Two fantastic goals and one quite wonderful assist swung a captivating Clasico against Real Madrid in Barcelona\u2019s favour . And yet , after the game , nobody was talking about the Argentine forward . There were other talking points , of course . There was Marcelo\u2019s foul on Cesc Fabregas , Jose Mourinho poking Barca assistant coach Tito Vilanova in the eye , David Villa\u2019s slap on Mesut Ozil and a whole host of other ill-tempered incidents , not to menti
 on an epic encounter between two superb sides . Nevertheless , Messi\u2019s masterclass had been the difference between a magnificent Madrid and a Barca team struggling to keep the pace with their biggest rivals . Having netted a vital goal in the first leg at the Santiago Bernabeu , Leo won the return almost single-handedly . It was a performance worthy of acclaim and accolades aplenty , of extreme eulogy . Quite simply , though , there is now little left to say about Messi\u2019s magic and marvel . As Pep Guardiola has oft opined : \u201c We are running out of words to describe Leo . \u201d Indeed , decisive displays have become the norm when it comes to this exceptional young man ; peerless performances are not only hoped for by Barcelona players and their fans , but expected . " There are no words to describe Messi . You have to see it - it is something you cannot describe because you have to see it to believe it . " - Barcelona boss Pep Guardiola The Camp Nou crowds have become accustomed 
 to special stars . Diego Maradona , Johan Cruyff , Ronaldo , Rivaldo and Romario have all graced the turf at Barca 's famous old stadium . But none have done so quite as brilliantly \u2013 or consistently \u2013 as Messi . The new season has barely begun and Leo already has three goals in two games , having only just returned from his holidays \u2013 and a disappointing Copa America campaign with Argentina \u2013 ahead of the Supercopa\u2019s first leg in Madrid . Last term , he hit 53 in just 55 games , racking up an incredible 24 assists along the way . MOMENT OF THE SEASON CHAMPIONS LEAGUE SEMI-FINAL L1 REAL MADRID 0-2 BARCELONA Just as he did in last week 's Spanish Supercopa , Messi proved the difference between Barca and Madrid as he decided this tie with a brilliant brace at the Bernabeu , including a stunning second which saw him beat no less than four Real players on a trademark slalom run towards goal and angled finish past Iker Casillas . Messi missed out as Barca began the 2010-11 camp
 aign with a 3-1 defeat in the first leg of the Supercopa in Seville , having been away on international duty earlier in the week . Back for the Camp Nou clash , however , the Argentine blew away the Andalusians with a brilliant hat-trick to ensure Barca started the season as they have become accostumed of late \u2013 by winning trophies . Leo then took just three minutes to make his mark in La Liga , netting his side\u2019s first of the campaign in a 3-0 win at Racing Santander . Be it Supercopa , Liga , Copa del Rey or Champions League , Messi made his mark . A brilliant brace against Panathinaikos was accompanied by two assists and the Argentine came within a whisker of his hat-trick , rattling the woodwork on two occasions . A treble did come in a 5-0 thrashing of Betis in the Copa del Rey , though , while another arrived in the historic 8-0 humiliation of Almeria . Those were part of the forward\u2019s most prolific scoring run as he netted in nine consecutive games . He was unable to ma
 ke it a perfect 10 in his side\u2019s next fixture , but will hardly have cared as the Catalans trounced Real Madrid 5-0 . In that match , Messi demonstrated to the watching world just how much of a complete player he has become . There were no goals , nor mazy runs , but two glorious assists for Villa and a breathtaking display of pressure and passing to inflict upon Mourinho his worst ever result as coach . The goals continued to fly in after that , with two more at Osasuna and another brace against Real Sociedad , capping a sensational 2010 as he beat off team-mates Andres Iniesta and Xavi to the Fifa Ballon d\u2019Or . But he would be even more decisive in 2011. When Barca were in need of inspiration , there was Messi to provide it . Two goals , including a sensational strike to open the scoring , saw the Catalan club overcome a 2-1 first-leg deficit in the last 16 of the Champions League against Arsenal , while a hat-trick against Atletico Madrid saw the Catalans achieve a 16th succe
 ssive victory , breaking a record held by Alfredo Di Stefano\u2019s brilliant Real side of the 1960s . He also bagged the only goal of the game in a vital league win at Valencia and surpassed his previous mark of 47 goals \u2013 which he had tied with Ronaldo the season before \u2013 with another strike in the Champions League quarter-final against Shakhtar Donetsk . The best , however , was yet to come . After the disappointment of losing the final of the Copa del Rey to Madrid , Messi soon erased the pain with both goals \u2013 including a stunning second \u2013 as Guardiola\u2019s side erased memories of that defeat with a 2-0 win at the Santiago Bernabeu , which sealed their passage - following a 1-1 draw at Camp Nou in the second leg - to the European showpiece against Manchester United at Wembley . " While Ronaldo remains selfish at times , Leo 's decision-making is impeccable ; he knows when to shoot , when to pass and even when to return , conserving his energies for quick , intuitive bursts 
 and sprints . The results are often devastasting . " For the second time in three years , Barca and United squared up in the final . And Messi , who had converted with a superb header in a 2-0 win in 2009 , beat Edwin Van der Sar this time with a thunderous left-footed drive which gave the Dutch keeper no time to react on its way in from outside the box . It turned out to be the game-winning goal and was a fitting end to a fairytale season for Barca 's talisman . His 53 strikes saw him tied with Cristiano Ronaldo at the end of the campaign , but while both players were lauded for their sensational scoring form , Messi\u2019s were more often decisive . The Argentine\u2019s assist total also shows he provided more than the Portuguese . While Ronaldo remains selfish at times , Leo\u2019s decision-making is impeccable ; he knows when to shoot , when to pass and even when to run , conserving his energies for quick , intuitive bursts and sprints . The results are often devastating : Lionel Andres 
 Messi is the best player in the world\u2019s best team , still the finest footballer on the planet and our overwhelming choice as Goal . com\u2019s prime performer from the last 12 months . With few adjectives left to describe his talents , perhaps the title of the Tina Turner track that blasts out of the tannoys before Barca\u2019s matches at Camp Nou is all that is required . Goal . com 50\u2019s worthy winner , Lionel Messi : Simply The Best . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/81NewsGueyeA_BlackPete_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/News/81NewsGueyeA_BlackPete_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/81NewsGueyeA_BlackPete_EN.txt.txt
new file mode 100644
index 0000000..21740cb
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/81NewsGueyeA_BlackPete_EN.txt.txt
@@ -0,0 +1,2 @@
+
+The Netherlands : A Holiday Season of Festivities , Costumes \u2026 and Racism ? In the winter season in The Netherlands a character named Zwarte Piet ( Black Pete ) accompanies Sinterklaas ( Saint Nicholas , the original inspiration for Santa Claus ) for a yearly feast that is celebrated on the evening of December 5 or morning of December 6 with sweets and presents for all good children . This traditional holiday rivals Christmas in importance . In recent years the role of Zwarte Piet has become part of a recurring debate in The Netherlands as some citizens take offense at holiday costumes with black painted faces . The story goes that the companions of Saint Nicholas are Moors who help carry the presents brought to children when he arrives by boat from Spain . The tradition continues to be popular , though some have been moved to protest against what they see as racist imagery . On November 12 , 2011 a protester wearing a t-shirt that said \u2018 Zwarte Piet is racisme \u2019 ( Black Pete 
 is racism ) was arrested in Dordrecht amidst accusations of police brutality . The t-shirt campaign has its own Tumblr blog with photos and a Facebook page with more than 800 followers . The blogger at Stuff Dutch People Like wrote in 2010 about the tradition of Zwarte Piet : You know it\u2019s that time of the year again in Holland , when you are greeted by some Dutch person on the street , whose face is painted completely black and is sporting an afro wig , bright red lips and a ridiculous clown-like costume . Sinterklaas and Zwarte Piet , The Hague , The Netherlands , November 2008 , by Zemistor ( CC-BY-ND ) Dutch graffiti artist and blogger BNE posts some photos of Zwarte Piet , and asks : Is The Dutch Holiday Of Sinterklaas\u2019s Tradition Of \u201c Black Pete \u201d Racist ? : This \u201c tradition \u201d has evolved throughout the years , partially due to increasing protests from groups that find these depictions offensive . Nowadays , it is claimed that the Black face is due to the fact that
  the helpers have gone through chimneys and as a result , their faces are covered in soot . What again , nobody can clearly explain , is what kind of soot leaves such a uniform and evenly spread residue . Or worse , why these \u201c chimney dwellers \u201d speak in a fake accent that parodies the Black population of the Dutch former colony of Suriname . Anthropologist and blogger Martijn de Koning of CLOSER explains in Jolly Black Servant \u2013 Tradition and Racism in the Netherlands : I dont expect a change in this tradition very soon . It should be clear however that Black Pete is a construction , and invention that has already changed in history . The current tradition has lost many of negative connotations which is partly positive but the negative side is that this makes the racism more hidden . Nevertheless , I think this Dutch tradition lends itself perfectly for teaching young children about racism , colonialism and religion throughout history . Maybe that would be a starting point f
 or some change in the future ? On travel website Off Track Planet , Anna Starostinetskaya gives this answer to the question , What the F*ck is Zwarte Piet ? : So is Pete a children\u2019s tale or a racist figure ? We promise no definitive answer exists . We\u2019re not saying this tradition is not objectifying black people in a racist way and it is understandable that Americans have the strongest feelings on the topic because Zwarte Piet is visually too close to what our racist roots look like . But Americans must also realize that our own history drives us to apply what we know about our own racist past on traditions that may not have anything in common but black face paint . Although it may be racist in some way , we cannot just superimpose our own racist history atop another country\u2019s tradition and say it\u2019s the same . Either way , we hope a happy medium exists that doesn\u2019t involve smurfs , midgets or complete Americanization of world traditions . Sinterklaas arrives by boat in Ar
 nhem , November 2011 , by Bas Boerman ( CC-BY-NC ) On the blog Tiger Beatdown , beneath Flavia\u2019s post \u201c If you protest racism during Black Face season in The Netherlands , you will be beaten up and arrested \u201d a comment by Elfe echoes the above : I read your post because I needed to understand why I do not find this tradition racist \u2026The \u201c slaves \u201d or \u201c helpers \u201d are you refer to them are not ridicule : these are pages not clowns and they are wearing nice clothes , they are not parading around half naked with a bone across their nostrils like some savages ( or like Josephine Baker and her banana skirt ) . \u2026 Like \u201c Tintin in the Congo \u201d the Zwarte Piets are a reminder of the past . \u2026 I know it is very insulting for Blacks in America to see White people with their face painted in black ( but it took me to live in the US to understand why : a period when black were not even allowed to play their own role in theater ) . \u2026Like the rappers who have decided to own 
 the N word we can just ignore this tradition if it annoys us , personally I could not care less . Being African I don\u2019t see the Zwarte Piets as Blacks ( they don\u2019t look like me or like any African I know ) \u2026 To feel insulted by them you really need to have a really poor self-esteem . Sorry for being politically incorrect \u2026 Written by Anna Gueye 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/82NewsLeM_OrbanGoldmanSachs_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/News/82NewsLeM_OrbanGoldmanSachs_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/82NewsLeM_OrbanGoldmanSachs_EN.txt.txt
new file mode 100644
index 0000000..b6f53e5
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/82NewsLeM_OrbanGoldmanSachs_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Hungary : \u201c We Voted For Orb�n , Not For Goldman Sachs \u201d Last Saturday , after several protests organized by citizen movements and opposition forces against the politics of the ruling Fidesz-KDNP government , Hungarians supporting those in power decided to express their opinion at a rally called Peace March . According to the Hungarian Interior Ministry 's report [ hu ] , some 400,000 people expressed their support for the government at the peaceful - and cheerful - event . \u2018 We are the Hungarian people and we stand for Orb�n 's government ! ' . Photo by Redjade , used with permission . Those who thought Fidesz-KDNP had lost the trust of the Hungarian citizens , drawing the consequence from the extensive foreign media coverage of the opposition protests in Budapest , were challenged this time by the fact that the government elected in 2010 with a two-thirds majority still enjoyed the support of many . Fidelitas , a youth group derived from Fidesz , shared some 360-degree pan
 oramic photos of the march . The protesters marched from Heroes ' Square to Kossuth Square by the Parliament , where brief speeches were delivered . The main organizers of the event were Zsolt Bayer , author of opinion pieces at the conservative daily Magyar H�rlap , G�bor Sz�les , a wealthy Hungarian entrepreneur and owner of Magyar H�rlap , and Andr�s Bencsik , editor-in-chief of Magyar Demokrata [ hu ] , also a Hungarian conservative daily . The right-leaning blog Mandiner has been very critical of the government recently , and , at first , their blogger , Dobray , who visited the Peace March , also had some doubts regarding the event [ hu ] : [ \u2026 ] Compared to what I had anticipated , the march came off even better : the mass of 400,000 ( probably fewer than that , the protest maths [ competition of whose protest had more attendees ] was started by Bencsik at Kossuth Square when he said , referring to a television report , that they were 1 million , which was evidently an
  unreal figure ) walked the distance , and , as no other options were listed on the program , no lame events happened . The puritan minimalism goes hand in hand with a portion of boredom well known from the first , eventless left-wing rallies . But it 's hard to pick at that . And that there were some groups with Arpad 's striped flags [ a symbol of the far right ] was not a big deal , we are used to that , they do n't do any trouble . We will worry about some Arpad stripes protesters in a mass of a couple of hundred thousands if the left wing expels from their community the comrades parading in the USSR and Che t-shirts . [ \u2026 ] The fact that describes the complex situation in Hungary best is that the government 's supporters oppose the talks and future agreements on the bailout from the EU and the IMF , while the opposition is in favor of reaching the agreements as soon as possible , in order to strengthen Hungary 's volatile economy . Pro-government protesters criticized EU/ECB/
 IMF for pressure on the government to take more bailout loans . Photo by Redjade , used with permission . Many protesters arrived from outside the capital . The blog of the city of �csa wrote [ hu ] about why they considered it important to participate in the march : People set off from almost every settlement of the country to express their solidarity with the government elected with the two-thirds majority , with its leader Viktor Orb�n and with everyone who has been attacked in the past days. The marchers stand up for the sovereignity of Hungary and stick to the achievements of democracy , they ca n't stand that foreign politicians , businessmen , banks are willing to administer their lives . [ \u2026 ] V�lem�nyvez�r pointed out [ hu ] that most of the protesters were elderly : [ \u2026 ] it was very striking that most of the marchers were aged 50 or older . They are the ones whose private pension savings were not taken away , almost none of them has a foreign currency loan , and 
 the government specifically tried to support them , through measures like the one-time 8-percent pension makeup or by implementing the institution of securing employment for older persons . [ \u2026 ] 'We voted for Orb�n and not for Goldman Sachs ' . Photo by Redjade , used with permission . Dobray hints at the rumours about paid protesters and organized travel to the rally location , the accusations raised by opposition members : [ \u2026 ] So now we are even , now really each and everyone has brought politics to the street . And it 's funny that at any sort of protest the actual side opposing the protesters tries every method to discredit the other 's event ; and tries to find those whose travel has been paid for , who were paid to come and who were cheated , etc. Everyone is generous when it 's about their protest , but if it 's about the other 's , they turn petty and suspicious . The neighbour 's lawn is always wilted . I also would be happy if the Peace March did n't get listed amo
 ng the ultimate arguments of Fidesz government allowing them to knock down all the opposing opinions . [ \u2026 ] Zolt�n Ruzsbaczky of Mos Maiorum blog published a guest post [ hu ] on Konzervat�rium blog , noting that the huge number of the pro-government supporters may signify the arrival of a new stage of democracy in Hungary , with a lot of people daring to stand up for their opinion : [ \u2026 ] Of course , this needs a government that applies this trust and successfully navigates the tempestuous sea of international politics and with its economic policy it sets Hungary on the track of growth . Besides this , one ca n't get by those masses who still oppose the politics of the government . We will learn only later what the long-term effects [ of this march ] will be , [ and whether there will be any ] . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/83NewsMendesFrancoJ_HaitiBeyondCapital_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/News/83NewsMendesFrancoJ_HaitiBeyondCapital_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/83NewsMendesFrancoJ_HaitiBeyondCapital_EN.txt.txt
new file mode 100644
index 0000000..4f0580c
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/83NewsMendesFrancoJ_HaitiBeyondCapital_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Haiti : Beyond the Capital Posted 17 January 2010 22:42 GMT Much of the coverage of the destruction in the earthquake 's aftermath has been focused in and around Haiti 's capital city , Port-au-Prince . But many other areas close to the \u2018 quake 's epicentre have also been affected , as bloggers are quick to point out \u2026 Jacmel , located about twenty-five miles south of Port-au-Prince , is \u201c stranded and increasingly desperate \u201d , according to Repeating Islands ' republication of an excerpt from \u201c the award-winning team of reporters from the Miami Herald : Residents of Jacmel , a quaint , historic Caribbean port city that suffered widespread damage and has been cut off from Port-au-Prince to the north , complain they have been forgotten . Four days after the quake struck Jacmel with equal force , they say they are still awaiting food , water , medical supplies and relief workers . Despite the blog 's discontent \u201c about the nature of the coverage of the earthquake in Haiti 
 on American television and other media \u201d , in another post it follows the Herald team as it reports on another area that is receiving little media attention , Carrefour : This town , which on Tuesday was the epicenter of the earthquake , is living in the epicenter of oblivion . Pwoje Espwa - Hope in Haiti , meanwhile , reports on the relief efforts taking place in Les Cayes : In contrast to the situation in PAP , the UN is guiding the relief efforts in les Cayes , and will be coordinating and providing a platform for the efforts of all the NGOs working in the area . There is no fuel left for purchase in Cayes and the UN has very little left . The UN folks are not sure when food and fuel will be delivered . All of us are nervous about this . There was a commercial flight on Tortugair this afternoon from Cap Haitian to Cayes , and they delivered a group of 8 orthopedic surgeons to work at the hospital . As the people arrive here from the destroyed capitol we will assist them in any 
 way we can . Some need money to go on to family on the coast or inland ; some require medical attention ; all are hungry and thirsty ; almost all need clothing and shoes along with personal hygiene items . A simple thing like letting this young woman use my cell phone to call her mother and tell her she was all right and in Cayes was momentous for her and her mother . Konbit Pou Ayiti says that \u201c Haiti KONPAY has been playing a critical role coordinating a rapid response to the crisis in both Jacmel and Port-au-Prince\u2026pursuing two major strategies \u201d : 1. Delivering immediate support to people on the ground in Jacmel and Port-au-Prince by coordinating the transport of supplies and volunteers . Carefully design volunteer interventions to avoid exacerbating the developing food and water shortages . 2. Encouraging the evacuation of Port-au-Prince and establishing the resources necessary to assist victims when they arrive in the countryside by assessing existing resources in outlyi
 ng areas and sending teams and equipment to clinics . The post goes on to quote a report \u201c from Amber Munger on the ground in Port-au-Prince \u201d : These are some details of the damage in Jacmel , which is a city of 34,000 : � 1,785 homes completely destroyed � 4410 homes partially destroyed � 87 commercial businesses destroyed � 54 schools destroyed � 24 hotels destroyed � 26 churches destroyed � 5730 families displaced � Death count approaching 3,000 , nearly 10 % of the population ( Reported by Gwenn Mangine , www . mangine . org ) Mangine also posts images from the ( severely damaged ) general hospital , with a further update on Sunday 16 : \u2026we noticed that the main pharmacy in town was open . And so we went in and bought them out of everything they had from the list \u2014 alcohol , hand sanitizer , peroxide , wound care items , meds \u2026 ( another truckload . ) Yesterday we were expecting a big shipment of supplies , but we got one box . Still \u2013 we rushed over to the 
 hospital with it . Mostly antibiotics and trauma care supplies \u2013 both were desperately needed . The doctors were thrilled . Pye 's in Haiti discusses the \u201c crazy busy \u201d situation at the local airstrip : We had a plane full of supplies ready to come , however the San Juan airport would not let the plane leave with the medical supplies \u2026 . We are hoping that flights start today of supplies and medicine . And Darren Tyler of Conduit Mission , who has been trying to send emergency supplies to Jacmel by boat , shares an update from a member of his organisation on the ground : The port can be used , cruise ships can come in there . We need help bad here is the city . What kind of supplies are on the boat ? How fast can they get here ? We are starting to feel people get frustrated and scared \u2026 . Updates are being posted regularly on Twitter . @melindayiti noted ( 15 January ) that \u201c Jacmel is a mess - we have planes and boats but US coordinators wo n't give us clearance to get 
 in ! \u201d And a few hours later added : \u201c 2 boats on the way , still no clearance to land plane w/critical medical teams \u201d . Meanwhile , @RescueJacmel , a new Twitter account , is attempting to ensure that international rescue efforts do not overlook the small city . Video bloggers are also chronicling their experiences , with clips from Les Cayes and Jacmel getting lots of attention on YouTube and other video sharing websites . The Cine Institute in Jacmel also posted eyewitness accounts of the earthquake . Lougou Corner is one of the blogs eager to supply information from their community : # We last communicated with Ginette last Thursday evening and she said that an exodus of people left Port-au-Prince already and came back to the provinces and rural areas . # We communicated via email with some ministries in Cayes and they reported that hospitals in Cayes are flooded with patients returning from Port and other areas affected in the provinces . We have seen firsthand in Lougou
  how an entire community has changed when residents have a say on the issues that affect their own lives . They have the best knowledge on what can and should be done to meet their pressing needs and bring lasting change in their community . And finally , from a U. S. -based Caribbean diaspora blogger , comes a stirring account of her friend 's quest to find his mother , most likely in a neighbourhood of Port-au-Prince : \u2018 As we came up on the block , I first walked right by the house because a good portion of it was totally demolished . The fitness center across the street was also completely demolished with a really strong smell coming from in between the bricks . When I asked people if they knew my mom , they shook their heads until I mentioned her by her nickname , Tita . And they were like , \u201c Oh , yeah , \u201d with joy in their eyes . \u201c She 's right there in the house next door . \u201d \u2018 I opened the door . Her back was to me . I tapped her on the shoulder . The surprise ,
  the tears , the hug so hard to explain . It was an unbelievable moment . She squeezed me so hard , crazy with joy . She paraded me down the street . \u201c Meet my fourth son . He came for me , \u201d she said . \u201c He came for me . \u201d \u2018 We can only hope for similar stories coming out of other affected areas . For more on the earthquake in Haiti , visit our Special Coverage page . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/84NewsMillerH_FrankensteinTradition_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/News/84NewsMillerH_FrankensteinTradition_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/84NewsMillerH_FrankensteinTradition_EN.txt.txt
new file mode 100644
index 0000000..4dfdba2
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/84NewsMillerH_FrankensteinTradition_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Understanding the Frankenstein Tradition Henry I. Miller 2010-11-03 PALO ALTO \u2013 \u201c It\u2019s alive , it\u2019s moving , it\u2019s alive ... IT\u2019S ALIVE ! \u201d So said Dr. Victor Frankenstein when his \u201c creation \u201d was complete . Researchers have long been fascinated with trying to create life , but mainly they have had to settle for crafting variations of living organisms via mutation or other techniques of genetic engineering . In May , researchers at the J. Craig Venter Institute , led by Venter himself , synthesized the genome of a bacterium from scratch using chemical building blocks , and inserted it into the cell of a different variety of bacteria . The new genetic information \u201c rebooted \u201d its host cell and got it to function , replicate , and take on the characteristics of the \u201c donor . \u201d In other words , a sort of synthetic organism had been created . Reactions in the scientific community ranged from \u201c slight novelty \u201d to \u201c looming apocalypse . \u201d The former is m
 ore apt : Venter\u2019s creation is evolutionary , not revolutionary . The goal of \u201c synthetic biology , \u201d as the field is known , is to move microbiology and cell biology closer to the approach of engineering , so that standardized parts can be mixed , matched , and assembled \u2013 just as off-the-shelf chassis , engines , transmissions , and so on can be combined to build a hot-rod . Achieving this goal could offer scientists unprecedented opportunities for innovation , and better enable them to craft bespoke microorganisms and plants that produce pharmaceuticals , clean up toxic wastes , and obtain ( or \u201c fix \u201d ) nitrogen from the air ( obviating the need for chemical fertilizers ) . During the past half-century , genetic engineers , using increasingly powerful and precise tools and resources , have achieved breakthroughs that are opening up new opportunities in a broad array of fields . The Venter lab\u2019s achievement builds on similar work that began decades ago . In 1967 , a
  research group from Stanford Medical School and Caltech demonstrated the infectiousness of the genome of a bacterial virus called \u03a6\u03a7174 , whose DNA had been synthesized with an enzyme using the intact viral DNA as a template , or blueprint . That feat was hailed as \u201c life in a test tube . \u201d In 2002 , a research group at the State University of New York , Stony Brook , created a functional , infectious poliovirus solely from basic , off-the-shelf chemical building blocks . Their only blueprint for engineering the genome was the known sequence of RNA ( which comprises the viral genome and is chemically very similar to DNA ) . Similar to the 1967 experiments , the infectious RNA was synthesized enzymatically . It was able to direct the synthesis of viral proteins in the absence of a natural template . Once again , scientists had , in effect , created life in a test tube . Venter\u2019s group did much the same thing in the recently reported research , except that they used chemical 
 synthesis instead of enzymes to make the DNA . But some of the hype that surrounded the publication of the ensuing article in the journal Nature was disproportionate . Along with the Venter paper , Nature published eight commentaries on the significance of the work . The \u201c real \u201d scientists were aware of the incremental nature of the work , and questioned whether the Venter group had created a genuine \u201c synthetic cell , \u201d while the social scientists tended to exaggerate the implications of the work . Mark Bedau , a professor of philosophy at Reed College , wrote that the technology\u2019s \u201c new powers create new responsibilities . Nobody can be sure about the consequences of making new forms of life , and we must expect the unexpected and the unintended . This calls for fundamental innovations in precautionary thinking and risk analysis . \u201d But , with increasing sophistication , genetic engineers using old and new techniques have been creating organisms with novel or enhanc
 ed properties for decades . Regulations and standards of good practice already effectively address organisms that may be pathogenic or that threaten the natural environment . ( If anything , these standards are excessively burdensome . ) On the other hand , Swiss bioengineer Martin Fussenegger correctly observed that the Venter achievement \u201c is a technical advance , not a conceptual one . \u201d Other scientists noted that the organism is really only \u201c semi-synthetic , \u201d because the synthetic DNA ( which comprises only about 1 % of the dry weight of the cell ) was introduced into a normal , or non-synthetic , bacterium . Understanding the history of synthetic biology is important , because recognizing the correct paradigm has critical implications for how governments regulate it , which in turn affects the potential application and diffusion of the technology . Thirty-five years ago , the US National Institutes of Health adopted overly risk-averse guidelines for research using re
 combinant DNA , or \u201c genetic engineering , \u201d techniques . Those guidelines , based on what has proved to be an idiosyncratic and largely invalid set of assumptions , sent a powerful message that scientists and the federal government were taking seriously speculative , exaggerated risk scenarios \u2013 a message that has afflicted the technology\u2019s development worldwide ever since . Synthetic biology offers the prospect of powerful new tools for research and development in innumerable fields . But its potential can be fulfilled only if regulatory oversight is based on science , sound risk analysis , and an appreciation of the mistakes of history . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/85NewsRabinovichI_IranNuclear_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/News/85NewsRabinovichI_IranNuclear_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/85NewsRabinovichI_IranNuclear_EN.txt.txt
new file mode 100644
index 0000000..b2e6a74
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/85NewsRabinovichI_IranNuclear_EN.txt.txt
@@ -0,0 +1,2 @@
+
+The Iranian Nuclear Threat Goes Global Itamar Rabinovich 2012-02-16 TEL AVIV \u2013 The current drive to prevent Iran from developing a nuclear arsenal reflects two important , and interrelated , changes . From Israel\u2019s perspective , these changes are to be welcomed , though its government must remain cautious about the country\u2019s own role . The first change is the escalation of efforts by the United States and its Western allies to abort the Iranian regime\u2019s nuclear quest . This was instigated in part by the International Atomic Energy Agency\u2019s finding in November 2011 that Iran is indeed developing a nuclear weapon , and that it is getting perilously close to crossing the \u201c red line \u201d \u2013 the point beyond which its progress could no longer be stopped . Moreover , the US and its allies understand that failure to take serious action might prompt Israel to launch its own unilateral military offensive . The second change is the perception that Iran\u2019s nuclear capacity would t
 hreaten not only Israel . In a speech to the Union for Reform Judaism in December , US President Barack Obama stated that \u201c another threat to the security of Israel , the US , and the world is Iran\u2019s nuclear program . \u201d But , by this February , Obama was saying of Iran that \u201c my number-one priority continues to be the security of the US , but also the security of Israel , and we continue to work in lockstep as we proceed to try to solve this \u2026 \u201d That choice of words was no accident ; rather , it was a sign that the US is changing tack when it comes to Iran . For more than a decade , the question \u201c Whose issue is it ? \u201d has been part of the policy debate about Iran\u2019s nuclear ambitions . Israel\u2019s former prime minister , Ariel Sharon , used to caution his colleagues against \u201c rushing to the head of the line \u201d on Iran . He argued that if Israel were to take the lead in sounding the alarm on Iran\u2019s nuclear ambitions , the issue would be perceived as yet another 
 \u201c Israeli problem . \u201d Indeed , Israel\u2019s critics were already arguing that this was another case of the tail wagging the dog \u2013 that Israel and its American lobby were trying to push the US into serving Israel\u2019s interests rather than its own . The most egregious examples of this view were statements made by the political scientists John Mearsheimer and Stephen Walt . In a paper published prior to the release of their much-debated book The Israel Lobby , they argued : \u201c \u2026 Iran\u2019s nuclear ambitions do not pose an existential threat to the US . If Washington could live with a nuclear Soviet Union , a nuclear China , and even a nuclear North Korea , then it can live with a nuclear Iran . And that is why the [ Israel ] Lobby must keep constant pressure on US politicians to confront Tehran . \u201d Israel\u2019s current prime minister , Benjamin Netanyahu , has been less worried than Sharon was about Israel\u2019s perceived role . He is too busy being directly engaged in the attempt t
 o eliminate the deadly threat that a nuclear-armed Iran would pose to the Jewish state . Prior to the 2009 election that brought him to power , Netanyahu campaigned on the Iranian danger , and his government made the issue its cardinal concern . Together with his defense minister , Ehud Barak , Netanyahu succeeded in persuading Obama and the rest of the world that Israel was preparing a military attack as a last resort , should the US and its allies fail to stop the Iranian program in time . That policy has been effective , but it has also drawn attention to Israel\u2019s influence on the Iran question . Curiously , this has not been held against Israel , at least not so far , partly because Obama and other leaders now regard Iran as a more serious threat , and therefore feel the need to take appropriate action . The international community must underscore that its members are acting in the service of their national interests , and not simply for Israel\u2019s sake . But their willingness
  to engage could wane , particularly if sanctions exact a high financial price or military action causes a large number of casualties . Israel would therefore be wise to remember Sharon\u2019s cautionary words , and reinforce its pressure on the US administration with a broader diplomatic campaign . Like it or not , Israel must urge the world to remember that Iran is everyone\u2019s problem . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/86NewsRian_IranCutsOil_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/News/86NewsRian_IranCutsOil_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/86NewsRian_IranCutsOil_EN.txt.txt
new file mode 100644
index 0000000..e728122
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/86NewsRian_IranCutsOil_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Iran Cuts Oil Exports to UK , France Iran has stopped oil exports to British and French companies , the Islamic republic 's oil ministry said on Sunday . A statement the ministry 's website that Iran , OPEC 's second biggest oil producer after Saudi Arabia , would sell oil to " new customers . " Iran 's English-language television station Press TV said the move was " in line with the decision to end crude exports to six European states . " The European Union said last month it would stop importing Iranian crude from July 1 in a bid to force Iran to agree to halt its nuclear program . Western powers suspect Iran of seeking to create a nuclear bomb but Tehran insists its program is peaceful . Iranian media reported on Wednesday that Iran had cut oil exports to the Netherlands , Greece , France , Portugal , Spain and Italy in response to the EU oil embargo , but the country 's oil ministry later denied this . The 27-nation bloc currently buys about 20 percent of Iran 's oil exports . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/87NewsRian_MedvedevDismisses_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/News/87NewsRian_MedvedevDismisses_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/87NewsRian_MedvedevDismisses_EN.txt.txt
new file mode 100644
index 0000000..f5e88a7
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/87NewsRian_MedvedevDismisses_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Medvedev dismisses EU criticism of Duma elections Russian President Dmitry Medvedev on Thursday said a European Parliament resolution calling for new State Duma elections \u201c means nothing . \u201d \u201c I have nothing to say on this resolution , because these are our elections . The European Parliament has no relation to them . They can comment on anything they want . I will not comment on their decisions as they mean nothing to me , \u201d the Russian president said at a joint press conference with European Commission President Jose Manuel Barroso and European Council President Herman Van Rompuy . Russia\u2019s parliamentary parties have severely criticized the European Parliament\u2019s resolution , labeling it interference in the country\u2019s domestic affairs , Medvedev added . They have also demanded that it stop such \u201c escapades . \u201d 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/88NewsWiki_KaradzicArrest_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/News/88NewsWiki_KaradzicArrest_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/88NewsWiki_KaradzicArrest_EN.txt.txt
new file mode 100644
index 0000000..c1c6136
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/News/88NewsWiki_KaradzicArrest_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Alleged war criminal Radovan Karad\u017ei\u0107 caught in Serbia Tuesday , July 22 , 2008 Radovan Karad\u017ei\u0107 Alleged war criminal Radovan Karad\u017ei\u0107 was caught yesterday in Serbia by Serbian security forces after almost 13 years on the run from authorities . Last night he was questioned by an inquisitor of the War Crimes Court in Belgrade . Karad\u017ei\u0107 has been accused by the International Criminal Tribunal for the former Yugoslavia of genocide , war crimes and crimes against humanity during the Bosnian War from 1992 to 1995. The Srebrenica massacre , in which about 8,000 Muslims were killed , is among the most serious of his alleged crimes . The massacre was categorised as genocide by both the International Court of Justice and the International Criminal Tribunal for the former Yugoslavia . Internationally , the arrest was unanimously welcomed . Along with former military chief Ratko Mladi\u0107 , Karad\u017ei\u0107 has been one of the most sought-after war criminals of the Balkan conflict . Described 
 by the BBC 's Kate Adie as a " smart , rather vain man " , his capture found him with a long white beard working in a clinic and practicing alternative medicine under the name Dragan Dabic . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Opac/89OpacStallman_FreeSoft_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Opac/89OpacStallman_FreeSoft_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Opac/89OpacStallman_FreeSoft_EN.txt.txt
new file mode 100644
index 0000000..cde10d6
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Opac/89OpacStallman_FreeSoft_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff The Free Software Definition We maintain this free software definition to show clearly what must be true about a particular software program for it to be considered free software . From time to time we revise this definition to clarify it . If you would like to review the changes we 've made , please see the History section below for more information . \u201c Free software \u201d is a matter of liberty , not price . To understand the concept , you should think of \u201c free \u201d as in \u201c free speech , \u201d not as in \u201c free beer . \u201d Free software is a matter of the users ' freedom to run , copy , distribute , study , change and improve the software . More precisely , it means that the program 's users have the four essential freedoms : \u2022 The freedom to run the program , for any purpose ( freedom 0 ) . \u2022 The freedom to study how the program works , and change it so it does your computing as you wish ( freedom 1 ) . Access to the source code is a precondition for this . \u2022 The free
 dom to redistribute copies so you can help your neighbor ( freedom 2 ) . \u2022 The freedom to distribute copies of your modified versions to others ( freedom 3 ) . By doing this you can give the whole community a chance to benefit from your changes . Access to the source code is a precondition for this . A program is free software if users have all of these freedoms . Thus , you should be free to redistribute copies , either with or without modifications , either gratis or charging a fee for distribution , to anyone anywhere . Being free to do these things means ( among other things ) that you do not have to ask or pay for permission to do so . You should also have the freedom to make modifications and use them privately in your own work or play , without even mentioning that they exist . If you do publish your changes , you should not be required to notify anyone in particular , or in any particular way . The freedom to run the program means the freedom for any kind of person or orga
 nization to use it on any kind of computer system , for any kind of overall job and purpose , without being required to communicate about it with the developer or any other specific entity . In this freedom , it is the user 's purpose that matters , not the developer 's purpose ; you as a user are free to run the program for your purposes , and if you distribute it to someone else , she is then free to run it for her purposes , but you are not entitled to impose your purposes on her . The freedom to redistribute copies must include binary or executable forms of the program , as well as source code , for both modified and unmodified versions . ( Distributing programs in runnable form is necessary for conveniently installable free operating systems . ) It is OK if there is no way to produce a binary or executable form for a certain program ( since some languages do n't support that feature ) , but you must have the freedom to redistribute such forms should you find or develop a way to
  make them . In order for freedoms 1 and 3 ( the freedom to make changes and the freedom to publish improved versions ) to be meaningful , you must have access to the source code of the program . Therefore , accessibility of source code is a necessary condition for free software . Obfuscated \u201c source code \u201d is not real source code and does not count as source code . Freedom 1 includes the freedom to use your changed version in place of the original . If the program is delivered in a product designed to run someone else 's modified versions but refuse to run yours \u2014 a practice known as \u201c tivoization \u201d or \u201c lockdown \u201d , or ( in its practitioners ' perverse terminology ) as \u201c secure boot \u201d \u2014 freedom 1 becomes a theoretical fiction rather than a practical freedom . This is not sufficient . In other words , these binaries are not free software even if the source code they are compiled from is free . One important way to modify a program is by merging in available free
  subroutines and modules . If the program 's license says that you cannot merge in a suitably licensed existing module \u2014 for instance , if it requires you to be the copyright holder of any code you add \u2014 then the license is too restrictive to qualify as free . Freedom 3 includes the freedom to release your modified versions as free software . A free license may also permit other ways of releasing them ; in other words , it does not have to be a copyleft license . However , a license that requires modified versions to be nonfree does not qualify as a free license . In order for these freedoms to be real , they must be permanent and irrevocable as long as you do nothing wrong ; if the developer of the software has the power to revoke the license , or retroactively change its terms , without your doing anything wrong to give cause , the software is not free . However , certain kinds of rules about the manner of distributing free software are acceptable , when they do n't conflict w
 ith the central freedoms . For example , copyleft ( very simply stated ) is the rule that when redistributing the program , you cannot add restrictions to deny other people the central freedoms . This rule does not conflict with the central freedoms ; rather it protects them . \u201c Free software \u201d does not mean \u201c noncommercial . \u201d A free program must be available for commercial use , commercial development , and commercial distribution . Commercial development of free software is no longer unusual ; such free commercial software is very important . You may have paid money to get copies of free software , or you may have obtained copies at no charge . But regardless of how you got your copies , you always have the freedom to copy and change the software , even to sell copies . Whether a change constitutes an improvement is a subjective matter . If your modifications are limited , in substance , to changes that someone else considers an improvement , that is not freedom . However
  , rules about how to package a modified version are acceptable , if they do n't substantively limit your freedom to release modified versions , or your freedom to make and use modified versions privately . Thus , it is acceptable for the license to require that you change the name of the modified version , remove a logo , or identify your modifications as yours . As long as these requirements are not so burdensome that they effectively hamper you from releasing your changes , they are acceptable ; you 're already making other changes to the program , so you wo n't have trouble making a few more . Rules that \u201c if you make your version available in this way , you must make it available in that way also \u201d can be acceptable too , on the same condition . An example of such an acceptable rule is one saying that if you have distributed a modified version and a previous developer asks for a copy of it , you must send one . ( Note that such a rule still leaves you the choice of whether 
 to distribute your version at all . ) Rules that require release of source code to the users for versions that you put into public use are also acceptable . In the GNU project , we use copyleft to protect these freedoms legally for everyone . But noncopylefted free software also exists . We believe there are important reasons why it is better to use copyleft , but if your program is noncopylefted free software , it is still basically ethical . ( See Categories of Free Software for a description of how \u201c free software , \u201d \u201c copylefted software \u201d and other categories of software relate to each other . ) Sometimes government export control regulations and trade sanctions can constrain your freedom to distribute copies of programs internationally . Software developers do not have the power to eliminate or override these restrictions , but what they can and must do is refuse to impose them as conditions of use of the program . In this way , the restrictions will not affect activi
 ties and people outside the jurisdictions of these governments . Thus , free software licenses must not require obedience to any export regulations as a condition of any of the essential freedoms . Most free software licenses are based on copyright , and there are limits on what kinds of requirements can be imposed through copyright . If a copyright-based license respects freedom in the ways described above , it is unlikely to have some other sort of problem that we never anticipated ( though this does happen occasionally ) . However , some free software licenses are based on contracts , and contracts can impose a much larger range of possible restrictions . That means there are many possible ways such a license could be unacceptably restrictive and nonfree . We ca n't possibly list all the ways that might happen . If a contract-based license restricts the user in an unusual way that copyright-based licenses cannot , and which is n't mentioned here as legitimate , we will have to th
 ink about it , and we will probably conclude it is nonfree . When talking about free software , it is best to avoid using terms like \u201c give away \u201d or \u201c for free , \u201d because those terms imply that the issue is about price , not freedom . Some common terms such as \u201c piracy \u201d embody opinions we hope you wo n't endorse . See Confusing Words and Phrases that are Worth Avoiding for a discussion of these terms . We also have a list of proper translations of \u201c free software \u201d into various languages . Finally , note that criteria such as those stated in this free software definition require careful thought for their interpretation . To decide whether a specific software license qualifies as a free software license , we judge it based on these criteria to determine whether it fits their spirit as well as the precise words . If a license includes unconscionable restrictions , we reject it , even if we did not anticipate the issue in these criteria . Sometimes a license requirem
 ent raises an issue that calls for extensive thought , including discussions with a lawyer , before we can decide if the requirement is acceptable . When we reach a conclusion about a new issue , we often update these criteria to make it easier to see why certain licenses do or do n't qualify . If you are interested in whether a specific license qualifies as a free software license , see our list of licenses . If the license you are concerned with is not listed there , you can ask us about it by sending us email at 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Opac/90OpacTeam_Berlin_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Opac/90OpacTeam_Berlin_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Opac/90OpacTeam_Berlin_EN.txt.txt
new file mode 100644
index 0000000..5b77713
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Opac/90OpacTeam_Berlin_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Berlin Declaration on Open Access to Knowledge in the Sciences and Humanities The Internet has fundamentally changed the practical and economic realities of distributing scientific knowledge and cultural heritage . For the first time ever , the Internet now offers the chance to constitute a global and interactive representation of human knowledge , including cultural heritage and the guarantee of worldwide access . We , the undersigned , feel obliged to address the challenges of the Internet as an emerging functional medium for distributing knowledge . Obviously , these developments will be able to significantly modify the nature of scientific publishing as well as the existing system of quality assurance . In accordance with the spirit of the Declaration of the Budapest Open Access Initiative , the ECHO Charter and the Bethesda Statement on Open Access Publishing , we have drafted the Berlin Declaration to promote the Internet as a functional instrument for a global scientific 
 knowledge base and human reflection and to specify measures which research policy makers , research institutions , funding agencies , libraries , archives and museums need to consider . Goals Our mission of disseminating knowledge is only half complete if the information is not made widely and readily available to society . New possibilities of knowledge dissemination not only through the classical form but also and increasingly through the open access paradigm via the Internet have to be supported . We define open access as a comprehensive source of human knowledge and cultural heritage that has been approved by the scientific community . In order to realize the vision of a global and accessible representation of knowledge , the future Web has to be sustainable , interactive , and transparent . Content and software tools must be openly accessible and compatible . Definition of an Open Access Contribution Establishing open access as a worthwhile procedure ideally requires the active
  commitment of each and every individual producer of scientific knowledge and holder of cultural heritage . Open access contributions include original scientific research results , raw data and metadata , source materials , digital representations of pictorial and graphical materials and scholarly multimedia material . Preface Open access contributions must satisfy two conditions : 1. The author(s ) and right holder(s ) of such contributions grant(s ) to all users a free , irrevocable , worldwide , right of access to , and a license to copy , use , distribute , transmit and display the work publicly and to make and distribute derivative works , in any digital medium for any responsible purpose , subject to proper attribution of authorship ( community standards , will continue to provide the mechanism for enforcement of proper attribution and responsible use of the published work , as they do now ) , as well as the right to make small numbers of printed copies for their personal use 
 . 2. A complete version of the work and all supplemental materials , including a copy of the permission as stated above , in an appropriate standard electronic format is deposited ( and thus published ) in at least one online repository using suitable technical standards ( such as the Open Archive definitions ) that is supported and maintained by an academic institution , scholarly society , government agency , or other well established organization that seeks to enable open access , unrestricted distribution , inter operability , and long-term archiving . Supporting the Transition to the Electronic Open Access Paradigm Our organizations are interested in the further promotion of the new open access paradigm to gain the most benefit for science and society . Therefore , we intend to make progress by � encouraging our researchers/grant recipients to publish their work according to the principles of the open access paradigm . � encouraging the holders of cultural heritage to support
  open access by providing their resources on the Internet . � developing means and ways to evaluate open access contributions and online journals in order to maintain the standards of quality assurance and good scientific practice . � advocating that open access publication be recognized in promotion and tenure evaluation . � advocating the intrinsic merit of contributions to an open access infrastructure by software tool development , content provision , metadata creation , or the publication of individual articles . We realize that the process of moving to open access changes the dissemination of knowledge with respect to legal and financial aspects . Our organizations aim to find solutions that support further development of the existing legal and financial frameworks in order to facilitate optimal use and access . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Opac/91OpacTeam_Budapest_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Opac/91OpacTeam_Budapest_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Opac/91OpacTeam_Budapest_EN.txt.txt
new file mode 100644
index 0000000..d91932b
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Opac/91OpacTeam_Budapest_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Budapest Open Access Initiative An old tradition and a new technology have converged to make possible an unprecedented public good . The old tradition is the willingness of scientists and scholars to publish the fruits of their research in scholarly journals without payment , for the sake of inquiry and knowledge . The new technology is the internet . The public good they make possible is the world-wide electronic distribution of the peer-reviewed journal literature and completely free and unrestricted access to it by all scientists , scholars , teachers , students , and other curious minds . Removing access barriers to this literature will accelerate research , enrich education , share the learning of the rich with the poor and the poor with the rich , make this literature as useful as it can be , and lay the foundation for uniting humanity in a common intellectual conversation and quest for knowledge . For various reasons , this kind of free and unrestricted online availabilit
 y , which we will call open access , has so far been limited to small portions of the journal literature . But even in these limited collections , many different initiatives have shown that open access is economically feasible , that it gives readers extraordinary power to find and make use of relevant literature , and that it gives authors and their works vast and measurable new visibility , readership , and impact . To secure these benefits for all , we call on all interested institutions and individuals to help open up access to the rest of this literature and remove the barriers , especially the price barriers , that stand in the way . The more who join the effort to advance this cause , the sooner we will all enjoy the benefits of open access . The literature that should be freely accessible online is that which scholars give to the world without expectation of payment . Primarily , this category encompasses their peer-reviewed journal articles , but it also includes any unrevi
 ewed preprints that they might wish to put online for comment or to alert colleagues to important research findings . There are many degrees and kinds of wider and easier access to this literature . By " open access " to this literature , we mean its free availability on the public internet , permitting any users to read , download , copy , distribute , print , search , or link to the full texts of these articles , crawl them for indexing , pass them as data to software , or use them for any other lawful purpose , without financial , legal , or technical barriers other than those inseparable from gaining access to the internet itself . The only constraint on reproduction and distribution , and the only role for copyright in this domain , should be to give authors control over the integrity of their work and the right to be properly acknowledged and cited . While the peer-reviewed journal literature should be accessible online without cost to readers , it is not costless to produce .
  However , experiments show that the overall costs of providing open access to this literature are far lower than the costs of traditional forms of dissemination . With such an opportunity to save money and expand the scope of dissemination at the same time , there is today a strong incentive for professional associations , universities , libraries , foundations , and others to embrace open access as a means of advancing their missions . Achieving open access will require new cost recovery models and financing mechanisms , but the significantly lower overall cost of dissemination is a reason to be confident that the goal is attainable and not merely preferable or utopian . To achieve open access to scholarly journal literature , we recommend two complementary strategies . I. Self-Archiving : First , scholars need the tools and assistance to deposit their refereed journal articles in open electronic archives , a practice commonly called , self-archiving . When these archives conform 
 to standards created by the Open Archives Initiative , then search engines and other tools can treat the separate archives as one . Users then need not know which archives exist or where they are located in order to find and make use of their contents . II . Open-access Journals : Second , scholars need the means to launch a new generation of journals committed to open access , and to help existing journals that elect to make the transition to open access . Because journal articles should be disseminated as widely as possible , these new journals will no longer invoke copyright to restrict access to and use of the material they publish . Instead they will use copyright and other tools to ensure permanent open access to all the articles they publish . Because price is a barrier to access , these new journals will not charge subscription or access fees , and will turn to other methods for covering their expenses . There are many alternative sources of funds for this purpose , includin
 g the foundations and governments that fund research , the universities and laboratories that employ researchers , endowments set up by discipline or institution , friends of the cause of open access , profits from the sale of add-ons to the basic texts , funds freed up by the demise or cancellation of journals charging traditional subscription or access fees , or even contributions from the researchers themselves . There is no need to favor one of these solutions over the others for all disciplines or nations , and no need to stop looking for other , creative alternatives . Open access to peer-reviewed journal literature is the goal . Self-archiving ( I. ) and a new generation of open-access journals ( II . ) are the ways to attain this goal . They are not only direct and effective means to this end , they are within the reach of scholars themselves , immediately , and need not wait on changes brought about by markets or legislation . While we endorse the two strategies just outlin
 ed , we also encourage experimentation with further ways to make the transition from the present methods of dissemination to open access . Flexibility , experimentation , and adaptation to local circumstances are the best ways to assure that progress in diverse settings will be rapid , secure , and long-lived . The Open Society Institute , the foundation network founded by philanthropist George Soros , is committed to providing initial help and funding to realize this goal . It will use its resources and influence to extend and promote institutional self-archiving , to launch new open-access journals , and to help an open-access journal system become economically self-sustaining . While the Open Society Institute 's commitment and resources are substantial , this initiative is very much in need of other organizations to lend their effort and resources . We invite governments , universities , libraries , journal editors , publishers , foundations , learned societies , professional as
 sociations , and individual scholars who share our vision to join us in the task of removing the barriers to open access and building a future in which research and education in every part of the world are that much more free to flourish . February 14 , 2002 Budapest , Hungary 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/100TediOConnellA_Quantum_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/100TediOConnellA_Quantum_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/100TediOConnellA_Quantum_EN.txt.txt
new file mode 100644
index 0000000..90ff9b3
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/100TediOConnellA_Quantum_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff This is a representation of your brain . And your brain can be broken into two parts . There 's the left half , which is the logical side , and then the right half , which is the intuitive . And so if we had a scale to measure the aptitude of each hemisphere , then we can plot our brain . And for example , this would be somebody who 's completely logical . This would be someone who 's entirely intuitive . So where would you put your brain on this scale ? Some of us may have opted for one of these extremes , but I think for most people in the audience , your brain is something like this -- with a high aptitude in both hemispheres at the same time . It 's not like they 're mutually exclusive or anything . You can be logical and intuitive . And so I consider myself one of these people , along with most of the other experimental quantum physicists , who need a good deal of logic to string together these complex ideas . But at the same time , we need a good deal of intuition to actua
 lly make the experiments work . How do we develop this intuition ? Well we like to play with stuff . So we go out and play with it , and then we see how it acts . And then we develop our intuition from there . And really you do the same thing . So some intuition that you may have developed over the years is that one thing is only in one place at a time . I mean , it can sound weird to think about one thing being in two different places at the same time , but you were n't born with this notion , you developed it . And I remember watching a kid playing on a car stop . He was just a toddler and he was n't very good at it , and he kept falling over . But I bet playing with this car stop taught him a really valuable lesson , and that 's that large things do n't let you get right past them , and that they stay in one place . And so this is a great conceptual model to have of the world , unless you 're a particle physicist . It 'd be a terrible model for a particle physicist , because they
  do n't play with car stops , they play with these little weird particles . And when they play with their particles , they find they do all sorts of really weird things -- like they can fly right through walls , or they can be in two different places at the same time . And so they wrote down all these observations , and they called it the theory of quantum mechanics . And so that 's where physics was at a few years ago ; you needed quantum mechanics to describe little , tiny particles . But you did n't need it to describe the large , everyday objects around us . This did n't really sit well with my intuition , and maybe it 's just because I do n't play with particles very often . Well , I play with them sometimes , but not very often . And I 've never seen them . I mean , nobody 's ever seen a particle . But it did n't sit well with my logical side either . Because if everything is made up of little particles and all the little particles follow quantum mechanics , then should n't ev
 erything just follow quantum mechanics ? I do n't see any reason why it should n't . And so I 'd feel a lot better about the whole thing if we could somehow show that an everyday object also follows quantum mechanics . So a few years ago , I set off to do just that . So I made one . This is the first object that you can see that has been in a mechanical quantum superposition . So what we 're looking at here is a tiny computer chip . And you can sort of see this green dot right in the middle . And that 's this piece of metal I 'm going to be talking about in a minute . This is a photograph of the object . And here I 'll zoom-in a little bit . We 're looking right there in the center . And then here 's a really , really big close-up of the little piece of metal . So what we 're looking at is a little chunk of metal , and it 's shaped like a diving board , and it 's sticking out over a ledge . And so I made this thing in nearly the same way as you make a computer chip . I went into a c
 lean room with a fresh silicon wafer , and then I just cranked away at all the big machines for about 100 hours . For the last stuff , I had to build my own machine -- to make this swimming pool-shaped hole underneath the device . This device has the ability to be in a quantum superposition , but it needs a little help to do it . Here , let me give you an analogy . You know how uncomfortable it is to be in a crowded elevator ? I mean , when I 'm in an elevator all alone , I do all sorts of weird things , but then other people get on board and I stop doing those things , because I do n't want to bother them , or , frankly , scare them . So quantum mechanics says that inanimate objects feel the same way . The fellow passengers for inanimate objects are not just people , but it 's also the light shining on it and the wind blowing past it and the heat of the room . And so we knew , if we wanted to see this piece of metal behave quantum mechanically , we 're going to have to kick out all
  the other passengers . And so that 's what we did . We turned off the lights , and then we put it in a vacuum and sucked out all the air , and then we cooled it down to just a fraction of a degree above absolute zero . Now , all alone in the elevator , the little chunk of metal is free to act however it wanted . And so we measured its motion . We found it was moving in really weird ways . Instead of just sitting perfectly still , it was vibrating . And the way it was vibrating was breathing something like this -- like expanding and contracting bellows . And by giving it a gentle nudge , we were able to make it both vibrate and not vibrate at the same time -- something that 's only allowed with quantum mechanics . So what I 'm telling you here is something truly fantastic . What does it mean for one thing to be both vibrating and not vibrating at the same time ? So let 's think about the atoms . So one case : all the trillions of atoms that make up that chunk of metal are sitting st
 ill and at the same time those same atoms are moving up and down . Now it 's only at precise times when they align . The rest of the time they 're delocalized . That means that every atom is in two different places at the same time , which in turn means the entire chunk of metal is in two different places . I think this is really cool . ( Laughter ) Really . ( Applause ) It was worth locking myself in a clean room to do this for all those years . Because , check this out , the difference in scale between a single atom and that chunk of metal is about the same as the difference between that chunk of metal and you . So if a single atom can be in two different places at the same time , that chunk of metal can be in two different places , then why not you ? I mean , this is just my logical side talking . So imagine if you 're in multiple places at the same time , what would that be like ? How would your consciousness handle your body being delocalized in space ? There 's one more part t
 o the story . It 's when we warmed it up , and we turned on the lights and looked inside the box , we saw that the piece metal was still there in one piece . And so I had to develop this new intuition , that it seems like all the objects in the elevator are really just quantum objects just crammed into a tiny space . You hear a lot of talk about how quantum mechanics says that everything is all interconnected . Well , that 's not quite right ; it 's more than that , it 's deeper . It 's that those connections , your connections to all the things around you , literally define who you are . And that 's the profound weirdness of quantum mechanics . Thank you . ( Applause ) 
\ No newline at end of file


[34/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/54FictDumasAp_3Musk_11_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/54FictDumasAp_3Musk_11_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/54FictDumasAp_3Musk_11_EN.txt.txt
new file mode 100644
index 0000000..0693c71
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/54FictDumasAp_3Musk_11_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff 11 IN WHICH THE PLOT THICKENS His visit to M. de Treville being paid , the pensive d'Artagnan took the longest way homeward . On what was d'Artagnan thinking , that he strayed thus from his path , gazing at the stars of heaven , and sometimes sighing , sometimes smiling ? He was thinking of Mme . Bonacieux . For an apprentice Musketeer the young woman was almost an ideal of love . Pretty , mysterious , initiated in almost all the secrets of the court , which reflected such a charming gravity over her pleasing features , it might be surmised that she was not wholly unmoved ; and this is an irresistible charm to novices in love . Moreover , d'Artagnan had delivered her from the hands of the demons who wished to search and ill treat her ; and this important service had established between them one of those sentiments of gratitude which so easily assume a more tender character . D'Artagnan already fancied himself , so rapid is the flight of our dreams upon the wings of imagination ,
  accosted by a messenger from the young woman , who brought him some billet appointing a meeting , a gold chain , or a diamond . We have observed that young cavaliers received presents from their king without shame . Let us add that in these times of lax morality they had no more delicacy with respect to the mistresses ; and that the latter almost always left them valuable and durable remembrances , as if they essayed to conquer the fragility of their sentiments by the solidity of their gifts . Without a blush , men made their way in the world by the means of women blushing . Such as were only beautiful gave their beauty , whence , without doubt , comes the proverb , " The most beautiful girl in the world can only give what she has . " Such as were rich gave in addition a part of their money ; and a vast number of heroes of that gallant period may be cited who would neither have won their spurs in the first place , nor their battles afterward , without the purse , more or less furni
 shed , which their mistress fastened to the saddle bow . D'Artagnan owned nothing . Provincial diffidence , that slight varnish , the ephemeral flower , that down of the peach , had evaporated to the winds through the little orthodox counsels which the three Musketeers gave their friend . D'Artagnan , following the strange custom of the times , considered himself at Paris as on a campaign , neither more nor less than if he had been in Flanders--Spain yonder , woman here . In each there was an enemy to contend with , and contributions to be levied . But , we must say , at the present moment d'Artagnan was ruled by a feeling much more noble and disinterested . The mercer had said that he was rich ; the young man might easily guess that with so weak a man as M. Bonacieux ; and interest was almost foreign to this commencement of love , which had been the consequence of it . We say ALMOST , for the idea that a young , handsome , kind , and witty woman is at the same time rich takes nothi
 ng from the beginning of love , but on the contrary strengthens it . There are in affluence a crowd of aristocratic cares and caprices which are highly becoming to beauty . A fine and white stocking , a silken robe , a lace kerchief , a pretty slipper on the foot , a tasty ribbon on the head do not make an ugly woman pretty , but they make a pretty woman beautiful , without reckoning the hands , which gain by all this ; the hands , among women particularly , to be beautiful must be idle . Then d'Artagnan , as the reader , from whom we have not concealed the state of his fortune , very well knows--d'Artagnan was not a millionaire ; he hoped to become one someday , but the time which in his own mind he fixed upon for this happy change was still far distant . In the meanwhile , how disheartening to see the woman one loves long for those thousands of nothings which constitute a woman 's happiness , and be unable to give her those thousands of nothings . At least , when the woman is rich
  and the lover is not , that which he cannot offer she offers to herself ; and although it is generally with her husband 's money that she procures herself this indulgence , the gratitude for it seldom reverts to him . Then d'Artagnan , disposed to become the most tender of lovers , was at the same time a very devoted friend , In the midst of his amorous projects for the mercer 's wife , he did not forget his friends . The pretty Mme . Bonacieux was just the woman to walk with in the Plain St. Denis or in the fair of St. Germain , in company with Athos , Porthos , and Aramis , to whom d'Artagnan had often remarked this . Then one could enjoy charming little dinners , where one touches on one side the hand of a friend , and on the other the foot of a mistress . Besides , on pressing occasions , in extreme difficulties , d'Artagnan would become the preserver of his friends . And M. Bonacieux ? whom d'Artagnan had pushed into the hands of the officers , denying him aloud although he ha
 d promised in a whisper to save him . We are compelled to admit to our readers that d'Artagnan thought nothing about him in any way ; or that if he did think of him , it was only to say to himself that he was very well where he was , wherever it might be . Love is the most selfish of all the passions . Let our readers reassure themselves . IF d'Artagnan forgets his host , or appears to forget him , under the pretense of not knowing where he has been carried , we will not forget him , and we know where he is . But for the moment , let us do as did the amorous Gascon ; we will see after the worthy mercer later . D'Artagnan , reflecting on his future amours , addressing himself to the beautiful night , and smiling at the stars , ascended the Rue Cherish-Midi , or Chase-Midi , as it was then called . As he found himself in the quarter in which Aramis lived , he took it into his head to pay his friend a visit in order to explain the motives which had led him to send Planchet with a reque
 st that he would come instantly to the mousetrap . Now , if Aramis had been at home when Planchet came to his abode , he had doubtless hastened to the Rue des Fossoyeurs , and finding nobody there but his other two companions perhaps , they would not be able to conceive what all this meant . This mystery required an explanation ; at least , so d'Artagnan declared to himself . He likewise thought this was an opportunity for talking about pretty little Mme . Bonacieux , of whom his head , if not his heart , was already full . We must never look for discretion in first love . First love is accompanied by such excessive joy that unless the joy be allowed to overflow , it will stifle you . Paris for two hours past had been dark , and seemed a desert . Eleven o'clock sounded from all the clocks of the Faubourg St. Germain . It was delightful weather . D'Artagnan was passing along a lane on the spot where the Rue d'Assas is now situated , breathing the balmy emanations which were borne upo
 n the wind from the Rue de Vaugirard , and which arose from the gardens refreshed by the dews of evening and the breeze of night . From a distance resounded , deadened , however , by good shutters , the songs of the tipplers , enjoying themselves in the cabarets scattered along the plain . Arrived at the end of the lane , d'Artagnan turned to the left . The house in which Aramis dwelt was situated between the Rue Cassette and the Rue Servandoni . D'Artagnan had just passed the Rue Cassette , and already perceived the door of his friend 's house , shaded by a mass of sycamores and clematis which formed a vast arch opposite the front of it , when he perceived something like a shadow issuing from the Rue Servandoni . This something was enveloped in a cloak , and d'Artagnan at first believed it was a man ; but by the smallness of the form , the hesitation of the walk , and the indecision of the step , he soon discovered that it was a woman . Further , this woman , as if not certain of t
 he house she was seeking , lifted up her eyes to look around her , stopped , went backward , and then returned again . D'Artagnan was perplexed . " Shall I go and offer her my services ? " thought he . " By her step she must be young ; perhaps she is pretty . Oh , yes ! But a woman who wanders in the streets at this hour only ventures out to meet her lover . If I should disturb a rendezvous , that would not be the best means of commencing an acquaintance . " Meantime the young woman continued to advance , counting the houses and windows . This was neither long nor difficult . There were but three hotels in this part of the street ; and only two windows looking toward the road , one of which was in a pavilion parallel to that which Aramis occupied , the other belonging to Aramis himself . " PARIDIEU ! " said d'Artagnan to himself , to whose mind the niece of the theologian reverted , " PARDIEU , it would be droll if this belated dove should be in search of our friend 's house . But o
 n my soul , it looks so . Ah , my dear Aramis , this time I shall find you out . " And d'Artagnan , making himself as small as he could , concealed himself in the darkest side of the street near a stone bench placed at the back of a niche . The young woman continued to advance ; and in addition to the lightness of her step , which had betrayed her , she emitted a little cough which denoted a sweet voice . D'Artagnan believed this cough to be a signal . Nevertheless , whether the cough had been answered by a similar signal which had fixed the irresolution of the nocturnal seeker , or whether without this aid she saw that she had arrived at the end of her journey , she resolutely drew near to Aramis 's shutter , and tapped , at three equal intervals , with her bent finger . " This is all very fine , dear Aramis , " murmured d'Artagnan . " Ah , Monsieur Hypocrite , I understand how you study theology . " The three blows were scarcely struck , when the inside blind was opened and a ligh
 t appeared through the panes of the outside shutter . " Ah , ah ! " said the listener , " not through doors , but through windows ! Ah , this visit was expected . We shall see the windows open , and the lady enter by escalade . Very pretty ! " But to the great astonishment of d'Artagnan , the shutter remained closed . Still more , the light which had shone for an instant disappeared , and all was again in obscurity . D'Artagnan thought this could not last long , and continued to look with all his eyes and listen with all his ears . He was right ; at the end of some seconds two sharp taps were heard inside . The young woman in the street replied by a single tap , and the shutter was opened a little way . It may be judged whether d'Artagnan looked or listened with avidity . Unfortunately the light had been removed into another chamber ; but the eyes of the young man were accustomed to the night . Besides , the eyes of the Gascons have , as it is asserted , like those of cats , the fac
 ulty of seeing in the dark . D'Artagnan then saw that the young woman took from her pocket a white object , which she unfolded quickly , and which took the form of a handkerchief . She made her interlocutor observe the corner of this unfolded object . This immediately recalled to d'Artagnan 's mind the handkerchief which he had found at the feet of Mme . Bonacieux , which had reminded him of that which he had dragged from under the feet of Aramis . " What the devil could that handkerchief signify ? " Placed where he was , d'Artagnan could not perceive the face of Aramis . We say Aramis , because the young man entertained no doubt that it was his friend who held this dialogue from the interior with the lady of the exterior . Curiosity prevailed over prudence ; and profiting by the preoccupation into which the sight of the handkerchief appeared to have plunged the two personages now on the scene , he stole from his hiding place , and quick as lightning , but stepping with utmost cauti
 on , he ran and placed himself close to the angle of the wall , from which his eye could pierce the interior of Aramis 's room . Upon gaining this advantage d'Artagnan was near uttering a cry of surprise ; it was not Aramis who was conversing with the nocturnal visitor , it was a woman ! D'Artagnan , however , could only see enough to recognize the form of her vestments , not enough to distinguish her features . At the same instant the woman inside drew a second handkerchief from her pocket , and exchanged it for that which had just been shown to her . Then some words were spoken by the two women . At length the shutter closed . The woman who was outside the window turned round , and passed within four steps of d'Artagnan , pulling down the hood of her mantle ; but the precaution was too late , d'Artagnan had already recognized Mme . Bonacieux . Mme . Bonacieux ! The suspicion that it was she had crossed the mind of d'Artagnan when she drew the handkerchief from her pocket ; but wha
 t probability was there that Mme . Bonacieux , who had sent for M. Laporte in order to be reconducted to the Louvre , should be running about the streets of Paris at half past eleven at night , at the risk of being abducted a second time ? This must be , then , an affair of importance ; and what is the most important affair to a woman of twenty-five ! Love . But was it on her own account , or on account of another , that she exposed herself to such hazards ? This was a question the young man asked himself , whom the demon of jealousy already gnawed , being in heart neither more nor less than an accepted lover . There was a very simple means of satisfying himself whither Mme . Bonacieux was going ; that was to follow her . This method was so simple that d'Artagnan employed it quite naturally and instinctively . But at the sight of the young man , who detached himself from the wall like a statue walking from its niche , and at the noise of the steps which she heard resound behind her 
 , Mme . Bonacieux uttered a little cry and fled . D'Artagnan ran after her . It was not difficult for him to overtake a woman embarrassed with her cloak . He came up with her before she had traversed a third of the street . The unfortunate woman was exhausted , not by fatigue , but by terror , and when d'Artagnan placed his hand upon her shoulder , she sank upon one knee , crying in a choking voice , " Kill me , if you please , you shall know nothing ! " D'Artagnan raised her by passing his arm round her waist ; but as he felt by her weight she was on the point of fainting , he made haste to reassure her by protestations of devotedness . These protestations were nothing for Mme . Bonacieux , for such protestations may be made with the worst intentions in the world ; but the voice was all . Mme . Bonacieux thought she recognized the sound of that voice ; she reopened her eyes , cast a quick glance upon the man who had terrified her so , and at once perceiving it was d'Artagnan , she 
 uttered a cry of joy , " Oh , it is you , it is you ! Thank God , thank God ! " " Yes , it is I , " said d'Artagnan , " it is I , whom God has sent to watch over you . " " Was it with that intention you followed me ? " asked the young woman , with a coquettish smile , whose somewhat bantering character resumed its influence , and with whom all fear had disappeared from the moment in which she recognized a friend in one she had taken for an enemy . " No , " said d'Artagnan ; " no , I confess it . It was chance that threw me in your way ; I saw a woman knocking at the window of one of my friends . " " One of your friends ? " interrupted Mme . Bonacieux . " Without doubt ; Aramis is one of my best friends . " " Aramis ! Who is he ? " " Come , come , you wo n't tell me you do n't know Aramis ? " " This is the first time I ever heard his name pronounced . " " It is the first time , then , that you ever went to that house ? " " Undoubtedly . " " And you did not know that it was inhabited 
 by a young man ? " " No. " " By a Musketeer ? " " No , indeed ! " " It was not he , then , you came to seek ? " " Not the least in the world . Besides , you must have seen that the person to whom I spoke was a woman . " " That is true ; but this woman is a friend of Aramis-- " " I know nothing of that . " " --since she lodges with him . " " That does not concern me . " " But who is she ? " " Oh , that is not my secret . " " My dear Madame Bonacieux , you are charming ; but at the same time you are one of the most mysterious women . " " Do I lose by that ? " " No ; you are , on the contrary , adorable . " " Give me your arm , then . " " Most willingly . And now ? " " Now escort me . " " Where ? " " Where I am going . " " But where are you going ? " " You will see , because you will leave me at the door . " " Shall I wait for you ? " " That will be useless . " " You will return alone , then ? " " Perhaps yes , perhaps no . " " But will the person who shall accompany you afterward be a
  man or a woman ? " " I do n't know yet . " " But I will know it ! " " How so ? " " I will wait until you come out . " " In that case , adieu . " " Why so ? " " I do not want you . " " But you have claimed-- " " The aid of a gentleman , not the watchfulness of a spy . " " The word is rather hard . " " How are they called who follow others in spite of them ? " " They are indiscreet . " " The word is too mild . " " Well , madame , I perceive I must do as you wish . " " Why did you deprive yourself of the merit of doing so at once ? " " Is there no merit in repentance ? " " And do you really repent ? " " I know nothing about it myself . But what I know is that I promise to do all you wish if you allow me to accompany you where you are going . " " And you will leave me then ? " " Yes . " " Without waiting for my coming out again ? " " Yes . " " Word of honor ? " " By the faith of a gentleman . Take my arm , and let us go . " D'Artagnan offered his arm to Mme . Bonacieux , who willingly 
 took it , half laughing , half trembling , and both gained the top of Rue de la Harpe . Arriving there , the young woman seemed to hesitate , as she had before done in the Rue Vaugirard . She seemed , however , by certain signs , to recognize a door , and approaching that door , " And now , monsieur , " said she , " it is here I have business ; a thousand thanks for your honorable company , which has saved me from all the dangers to which , alone I was exposed . But the moment is come to keep your word ; I have reached my destination . " " And you will have nothing to fear on your return ? " " I shall have nothing to fear but robbers . " " And that is nothing ? " " What could they take from me ? I have not a penny about me . " " You forget that beautiful handkerchief with the coat of arms . " " Which ? " " That which I found at your feet , and replaced in your pocket . " " Hold your tongue , imprudent man ! Do you wish to destroy me ? " " You see very plainly that there is still dan
 ger for you , since a single word makes you tremble ; and you confess that if that word were heard you would be ruined . Come , come , madame ! " cried d'Artagnan , seizing her hands , and surveying her with an ardent glance , " come , be more generous . Confide in me . Have you not read in my eyes that there is nothing but devotion and sympathy in my heart ? " " Yes , " replied Mme . Bonacieux ; " therefore , ask my own secrets , and I will reveal them to you ; but those of others--that is quite another thing . " " Very well , " said d'Artagnan , " I shall discover them ; as these secrets may have an influence over your life , these secrets must become mine . " " Beware of what you do ! " cried the young woman , in a manner so serious as to make d'Artagnan start in spite of himself . " Oh , meddle in nothing which concerns me . Do not seek to assist me in that which I am accomplishing . This I ask of you in the name of the interest with which I inspire you , in the name of the serv
 ice you have rendered me and which I never shall forget while I have life . Rather , place faith in what I tell you . Have no more concern about me ; I exist no longer for you , any more than if you had never seen me . " " Must Aramis do as much as I , madame ? " said d'Artagnan , deeply piqued . " This is the second or third time , monsieur , that you have repeated that name , and yet I have told you that I do not know him . " " You do not know the man at whose shutter you have just knocked ? Indeed , madame , you believe me too credulous ! " " Confess that it is for the sake of making me talk that you invent this story and create this personage . " " I invent nothing , madame ; I create nothing . I only speak that exact truth . " " And you say that one of your friends lives in that house ? " " I say so , and I repeat it for the third time ; that house is one inhabited by my friend , and that friend is Aramis . " " All this will be cleared up at a later period , " murmured the youn
 g woman ; " no , monsieur , be silent . " " If you could see my heart , " said d'Artagnan , " you would there read so much curiosity that you would pity me and so much love that you would instantly satisfy my curiosity . We have nothing to fear from those who love us . " " You speak very suddenly of love , monsieur , " said the young woman , shaking her head . " That is because love has come suddenly upon me , and for the first time ; and because I am only twenty . " The young woman looked at him furtively . " Listen ; I am already upon the scent , " resumed d'Artagnan . " About three months ago I was near having a duel with Aramis concerning a handkerchief resembling the one you showed to the woman in his house--for a handkerchief marked in the same manner , I am sure . " " Monsieur , " said the young woman , " you weary me very much , I assure you , with your questions . " " But you , madame , prudent as you are , think , if you were to be arrested with that handkerchief , and tha
 t handkerchief were to be seized , would you not be compromised ? " " In what way ? The initials are only mine--C . B. , Constance Bonacieux . " " Or Camille de Bois-Tracy . " " Silence , monsieur ! Once again , silence ! Ah , since the dangers I incur on my own account cannot stop you , think of those you may yourself run ! " " Me ? " " Yes ; there is peril of imprisonment , risk of life in knowing me . " " Then I will not leave you . " " Monsieur ! " said the young woman , supplicating him and clasping her hands together , " monsieur , in the name of heaven , by the honor of a soldier , by the courtesy of a gentleman , depart ! There , there midnight sounds ! That is the hour when I am expected . " " Madame , " said the young man , bowing ; " I can refuse nothing asked of me thus . Be content ; I will depart . " " But you will not follow me ; you will not watch me ? " " I will return home instantly . " " Ah , I was quite sure you were a good and brave young man , " said Mme . Bona
 cieux , holding out her hand to him , and placing the other upon the knocker of a little door almost hidden in the wall . D'Artagnan seized the hand held out to him , and kissed it ardently . " Ah ! I wish I had never seen you ! " cried d'Artagnan , with that ingenuous roughness which women often prefer to the affectations of politeness , because it betrays the depths of the thought and proves that feeling prevails over reason . " Well ! " resumed Mme . Bonacieux , in a voice almost caressing , and pressing the hand of d'Artagnan , who had not relinquished hers , " well : I will not say as much as you do ; what is lost for today may not be lost forever . Who knows , when I shall be at liberty , that I may not satisfy your curiosity ? " " And will you make the same promise to my love ? " cried d'Artagnan , beside himself with joy . " Oh , as to that , I do not engage myself . That depends upon the sentiments with which you may inspire me . " " Then today , madame-- " " Oh , today , I
  am no further than gratitude . " " Ah ! You are too charming , " said d'Artagnan , sorrowfully ; " and you abuse my love . " " No , I use your generosity , that 's all . But be of good cheer ; with certain people , everything comes round . " " Oh , you render me the happiest of men ! Do not forget this evening--do not forget that promise . " " Be satisfied . In the proper time and place I will remember everything . Now then , go , go , in the name of heaven ! I was expected at sharp midnight , and I am late . " " By five minutes . " " Yes ; but in certain circumstances five minutes are five ages . " " When one loves . " " Well ! And who told you I had no affair with a lover ? " " It is a man , then , who expects you ? " cried d'Artagnan . " A man ! " " The discussion is going to begin again ! " said Mme . Bonacieux , with a half-smile which was not exempt from a tinge of impatience . " No , no ; I go , I depart ! I believe in you , and I would have all the merit of my devotion , ev
 en if that devotion were stupidity . Adieu , madame , adieu ! " And as if he only felt strength to detach himself by a violent effort from the hand he held , he sprang away , running , while Mme . Bonacieux knocked , as at the shutter , three light and regular taps . When he had gained the angle of the street , he turned . The door had been opened , and shut again ; the mercer 's pretty wife had disappeared . D'Artagnan pursued his way . He had given his word not to watch Mme . Bonacieux , and if his life had depended upon the spot to which she was going or upon the person who should accompany her , d'Artagnan would have returned home , since he had so promised . Five minutes later he was in the Rue des Fossoyeurs . " Poor Athos ! " said he ; " he will never guess what all this means . He will have fallen asleep waiting for me , or else he will have returned home , where he will have learned that a woman had been there . A woman with Athos ! After all , " continued d'Artagnan , " th
 ere was certainly one with Aramis . All this is very strange ; and I am curious to know how it will end . " " Badly , monsieur , badly ! " replied a voice which the young man recognized as that of Planchet ; for , soliloquizing aloud , as very preoccupied people do , he had entered the alley , at the end of which were the stairs which led to his chamber . " How badly ? What do you mean by that , you idiot ? " asked d'Artagnan . " What has happened ? " " All sorts of misfortunes . " " What ? " " In the first place , Monsieur Athos is arrested . " " Arrested ! Athos arrested ! What for ? " " He was found in your lodging ; they took him for you . " " And by whom was he arrested ? " " By Guards brought by the men in black whom you put to flight . " " Why did he not tell them his name ? Why did he not tell them he knew nothing about this affair ? " " He took care not to do so , monsieur ; on the contrary , he came up to me and said , 'It is your master that needs his liberty at this mome
 nt and not I , since he knows everything and I know nothing . They will believe he is arrested , and that will give him time ; in three days I will tell them who I am , and they cannot fail to let me go . ' " " Bravo , Athos ! Noble heart ! " murmured d'Artagnan . " I know him well there ! And what did the officers do ? " " Four conveyed him away , I do n't know where--to the Bastille or Fort l'Eveque . Two remained with the men in black , who rummaged every place and took all the papers . The last two mounted guard at the door during this examination ; then , when all was over , they went away , leaving the house empty and exposed . " " And Porthos and Aramis ? " " I could not find them ; they did not come . " " But they may come any moment , for you left word that I awaited them ? " " Yes , monsieur . " " Well , do n't budge , then ; if they come , tell them what has happened . Let them wait for me at the Pomme-de-Pin . Here it would be dangerous ; the house may be watched . I wil
 l run to Monsieur de Treville to tell them all this , and will meet them there . " " Very well , monsieur , " said Planchet . " But you will remain ; you are not afraid ? " said d'Artagnan , coming back to recommend courage to his lackey . " Be easy , monsieur , " said Planchet ; " you do not know me yet . I am brave when I set about it . It is all in beginning . Besides , I am a Picard . " " Then it is understood , " said d'Artagnan ; " you would rather be killed than desert your post ? " " Yes , monsieur ; and there is nothing I would not do to prove to Monsieur that I am attached to him . " " Good ! " said d'Artagnan to himself . " It appears that the method I have adopted with this boy is decidedly the best . I shall use it again upon occasion . " And with all the swiftness of his legs , already a little fatigued however , with the perambulations of the day , d'Artagnan directed his course toward M. de Treville 's . M. de Treville was not at his hotel . His company was on guard 
 at the Louvre ; he was at the Louvre with his company . It was necessary to reach M. de Treville ; it was important that he should be informed of what was passing . D'Artagnan resolved to try and enter the Louvre . His costume of Guardsman in the company of M. Dessessart ought to be his passport . He therefore went down the Rue des Petits Augustins , and came up to the quay , in order to take the New Bridge . He had at first an idea of crossing by the ferry ; but on gaining the riverside , he had mechanically put his hand into his pocket , and perceived that he had not wherewithal to pay his passage . As he gained the top of the Rue Guenegaud , he saw two persons coming out of the Rue Dauphine whose appearance very much struck him . Of the two persons who composed this group , one was a man and the other a woman . The woman had the outline of Mme . Bonacieux ; the man resembled Aramis so much as to be mistaken for him . Besides , the woman wore that black mantle which d'Artagnan cou
 ld still see outlined on the shutter of the Rue de Vaugirard and on the door of the Rue de la Harpe ; still further , the man wore the uniform of a Musketeer . The woman 's hood was pulled down , and the man held a handkerchief to his face . Both , as this double precaution indicated , had an interest in not being recognized . They took the bridge . That was d'Artagnan 's road , as he was going to the Louvre . D'Artagnan followed them . He had not gone twenty steps before he became convinced that the woman was really Mme . Bonacieux and that the man was Aramis . He felt at that instant all the suspicions of jealousy agitating his heart . He felt himself doubly betrayed , by his friend and by her whom he already loved like a mistress . Mme . Bonacieux had declared to him , by all the gods , that she did not know Aramis ; and a quarter of an hour after having made this assertion , he found her hanging on the arm of Aramis . D'Artagnan did not reflect that he had only known the mercer 
 's pretty wife for three hours ; that she owed him nothing but a little gratitude for having delivered her from the men in black , who wished to carry her off , and that she had promised him nothing . He considered himself an outraged , betrayed , and ridiculed lover . Blood and anger mounted to his face ; he was resolved to unravel the mystery . The young man and young woman perceived they were watched , and redoubled their speed . D'Artagnan determined upon his course . He passed them , then returned so as to meet them exactly before the Samaritaine . Which was illuminated by a lamp which threw its light over all that part of the bridge . D'Artagnan stopped before them , and they stopped before him . " What do you want , monsieur ? " demanded the Musketeer , recoiling a step , and with a foreign accent , which proved to d'Artagnan that he was deceived in one of his conjectures . " It is not Aramis ! " cried he . " No , monsieur , it is not Aramis ; and by your exclamation I percei
 ve you have mistaken me for another , and pardon you . " " You pardon me ? " cried d'Artagnan . " Yes , " replied the stranger . " Allow me , then , to pass on , since it is not with me you have anything to do . " " You are right , monsieur , it is not with you that I have anything to do ; it is with Madame . " " With Madame ! You do not know her , " replied the stranger . " You are deceived , monsieur ; I know her very well . " " Ah , " said Mme . Bonacieux ; in a tone of reproach , " ah , monsieur , I had your promise as a soldier and your word as a gentleman . I hoped to be able to rely upon that . " " And I , madame ! " said d'Artagnan , embarrassed ; " you promised me-- " " Take my arm , madame , " said the stranger , " and let us continue our way . " D'Artagnan , however , stupefied , cast down , annihilated by all that happened , stood , with crossed arms , before the Musketeer and Mme . Bonacieux . The Musketeer advanced two steps , and pushed d'Artagnan aside with his hand 
 . D'Artagnan made a spring backward and drew his sword . At the same time , and with the rapidity of lightning , the stranger drew his . " In the name of heaven , my Lord ! " cried Mme . Bonacieux , throwing herself between the combatants and seizing the swords with her hands . " My Lord ! " cried d'Artagnan , enlightened by a sudden idea , " my Lord ! Pardon me , monsieur , but you are not-- " " My Lord the Duke of Buckingham , " said Mme . Bonacieux , in an undertone ; " and now you may ruin us all . " " My Lord , Madame , I ask a hundred pardons ! But I love her , my Lord , and was jealous . You know what it is to love , my Lord . Pardon me , and then tell me how I can risk my life to serve your Grace ? " " You are a brave young man , " said Buckingham , holding out his hand to d'Artagnan , who pressed it respectfully . " You offer me your services ; with the same frankness I accept them . Follow us at a distance of twenty paces , as far as the Louvre , and if anyone watches us ,
  slay him ! " D'Artagnan placed his naked sword under his arm , allowed the duke and Mme . Bonacieux to take twenty steps ahead , and then followed them , ready to execute the instructions of the noble and elegant minister of Charles I. Fortunately , he had no opportunity to give the duke this proof of his devotion , and the young woman and the handsome Musketeer entered the Louvre by the wicket of the Echelle without any interference . As for d'Artagnan , he immediately repaired to the cabaret of the Pomme-de-Pin , where he found Porthos and Aramis awaiting him . Without giving them any explanation of the alarm and inconvenience he had caused them , he told them that he had terminated the affair alone in which he had for a moment believed he should need their assistance . Meanwhile , carried away as we are by our narrative , we must leave our three friends to themselves , and follow the Duke of Buckingham and his guide through the labyrinths of the Louvre . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/55FictDumasAp_3Musk_2_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/55FictDumasAp_3Musk_2_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/55FictDumasAp_3Musk_2_EN.txt.txt
new file mode 100644
index 0000000..1877848
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/55FictDumasAp_3Musk_2_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff 2 THE ANTECHAMBER OF M. DE TREVILLE M. de Troisville , as his family was still called in Gascony , or M. de Treville , as he has ended by styling himself in Paris , had really commenced life as d'Artagnan now did ; that is to say , without a sou in his pocket , but with a fund of audacity , shrewdness , and intelligence which makes the poorest Gascon gentleman often derive more in his hope from the paternal inheritance than the richest Perigordian or Berrichan gentleman derives in reality from his . His insolent bravery , his still more insolent success at a time when blows poured down like hail , had borne him to the top of that difficult ladder called Court Favor , which he had climbed four steps at a time . He was the friend of the king , who honored highly , as everyone knows , the memory of his father , Henry IV . The father of M. de Treville had served him so faithfully in his wars against the league that in default of money--a thing to which the Bearnais was accustomed al
 l his life , and who constantly paid his debts with that of which he never stood in need of borrowing , that is to say , with ready wit--in default of money , we repeat , he authorized him , after the reduction of Paris , to assume for his arms a golden lion passant upon gules , with the motto FIDELIS ET FORTIS . This was a great matter in the way of honor , but very little in the way of wealth ; so that when the illustrious companion of the great Henry died , the only inheritance he was able to leave his son was his sword and his motto . Thanks to this double gift and the spotless name that accompanied it , M. de Treville was admitted into the household of the young prince where he made such good use of his sword , and was so faithful to his motto , that Louis XIII , one of the good blades of his kingdom , was accustomed to say that if he had a friend who was about to fight , he would advise him to choose as a second , himself first , and Treville next--or even , perhaps , before h
 imself . Thus Louis XIII had a real liking for Treville--a royal liking , a self-interested liking , it is true , but still a liking . At that unhappy period it was an important consideration to be surrounded by such men as Treville . Many might take for their device the epithet STRONG , which formed the second part of his motto , but very few gentlemen could lay claim to the FAITHFUL , which constituted the first . Treville was one of these latter . His was one of those rare organizations , endowed with an obedient intelligence like that of the dog ; with a blind valor , a quick eye , and a prompt hand ; to whom sight appeared only to be given to see if the king were dissatisfied with anyone , and the hand to strike this displeasing personage , whether a Besme , a Maurevers , a Poltiot de Mere , or a Vitry . In short , up to this period nothing had been wanting to Treville but opportunity ; but he was ever on the watch for it , and he faithfully promised himself that he would not f
 ail to seize it by its three hairs whenever it came within reach of his hand . At last Louis XIII made Treville the captain of his Musketeers , who were to Louis XIII in devotedness , or rather in fanaticism , what his Ordinaries had been to Henry III , and his Scotch Guard to Louis XI . On his part , the cardinal was not behind the king in this respect . When he saw the formidable and chosen body with which Louis XIII had surrounded himself , this second , or rather this first king of France , became desirous that he , too , should have his guard . He had his Musketeers therefore , as Louis XIII had his , and these two powerful rivals vied with each other in procuring , not only from all the provinces of France , but even from all foreign states , the most celebrated swordsmen . It was not uncommon for Richelieu and Louis XIII to dispute over their evening game of chess upon the merits of their servants . Each boasted the bearing and the courage of his own people . While exclaiming
  loudly against duels and brawls , they excited them secretly to quarrel , deriving an immoderate satisfaction or genuine regret from the success or defeat of their own combatants . We learn this from the memoirs of a man who was concerned in some few of these defeats and in many of these victories . Treville had grasped the weak side of his master ; and it was to this address that he owed the long and constant favor of a king who has not left the reputation behind him of being very faithful in his friendships . He paraded his Musketeers before the Cardinal Armand Duplessis with an insolent air which made the gray moustache of his Eminence curl with ire . Treville understood admirably the war method of that period , in which he who could not live at the expense of the enemy must live at the expense of his compatriots . His soldiers formed a legion of devil-may-care fellows , perfectly undisciplined toward all but himself . Loose , half-drunk , imposing , the king 's Musketeers , or 
 rather M. de Treville 's , spread themselves about in the cabarets , in the public walks , and the public sports , shouting , twisting their mustaches , clanking their swords , and taking great pleasure in annoying the Guards of the cardinal whenever they could fall in with them ; then drawing in the open streets , as if it were the best of all possible sports ; sometimes killed , but sure in that case to be both wept and avenged ; often killing others , but then certain of not rotting in prison , M. de Treville being there to claim them . Thus M. de Treville was praised to the highest note by these men , who adored him , and who , ruffians as they were , trembled before him like scholars before their master , obedient to his least word , and ready to sacrifice themselves to wash out the smallest insult . M. de Treville employed this powerful weapon for the king , in the first place , and the friends of the king--and then for himself and his own friends . For the rest , in the memoi
 rs of this period , which has left so many memoirs , one does not find this worthy gentleman blamed even by his enemies ; and he had many such among men of the pen as well as among men of the sword . In no instance , let us say , was this worthy gentleman accused of deriving personal advantage from the cooperation of his minions . Endowed with a rare genius for intrigue which rendered him the equal of the ablest intriguers , he remained an honest man . Still further , in spite of sword thrusts which weaken , and painful exercises which fatigue , he had become one of the most gallant frequenters of revels , one of the most insinuating lady 's men , one of the softest whisperers of interesting nothings of his day ; the BONNES FORTUNES of de Treville were talked of as those of M. de Bassompierre had been talked of twenty years before , and that was not saying a little . The captain of the Musketeers was therefore admired , feared , and loved ; and this constitutes the zenith of human f
 ortune . Louis XIV absorbed all the smaller stars of his court in his own vast radiance ; but his father , a sun PLURIBUS IMPAR , left his personal splendor to each of his favorites , his individual value to each of his courtiers . In addition to the leeves of the king and the cardinal , there might be reckoned in Paris at that time more than two hundred smaller but still noteworthy leeves . Among these two hundred leeves , that of Treville was one of the most sought . The court of his hotel , situated in the Rue du Vieux-Colombier , resembled a camp from by six o'clock in the morning in summer and eight o'clock in winter . From fifty to sixty Musketeers , who appeared to replace one another in order always to present an imposing number , paraded constantly , armed to the teeth and ready for anything . On one of those immense staircases , upon whose space modern civilization would build a whole house , ascended and descended the office seekers of Paris , who ran after any sort of fa
 vor--gentlemen from the provinces anxious to be enrolled , and servants in all sorts of liveries , bringing and carrying messages between their masters and M. de Treville . In the antechamber , upon long circular benches , reposed the elect ; that is to say , those who were called . In this apartment a continued buzzing prevailed from morning till night , while M. de Treville , in his office contiguous to this antechamber , received visits , listened to complaints , gave his orders , and like the king in his balcony at the Louvre , had only to place himself at the window to review both his men and arms . The day on which d'Artagnan presented himself the assemblage was imposing , particularly for a provincial just arriving from his province . It is true that this provincial was a Gascon ; and that , particularly at this period , the compatriots of d'Artagnan had the reputation of not being easily intimidated . When he had once passed the massive door covered with long square-headed n
 ails , he fell into the midst of a troop of swordsmen , who crossed one another in their passage , calling out , quarreling , and playing tricks one with another . In order to make one 's way amid these turbulent and conflicting waves , it was necessary to be an officer , a great noble , or a pretty woman . It was , then , into the midst of this tumult and disorder that our young man advanced with a beating heat , ranging his long rapier up his lanky leg , and keeping one hand on the edge of his cap , with that half-smile of the embarrassed a provincial who wishes to put on a good face . When he had passed one group he began to breathe more freely ; but he could not help observing that they turned round to look at him , and for the first time in his life d'Artagnan , who had till that day entertained a very good opinion of himself , felt ridiculous . Arrived at the staircase , it was still worse . There were four Musketeers on the bottom steps , amusing themselves with the following
  exercise , while ten or twelve of their comrades waited upon the landing place to take their turn in the sport . One of them , stationed upon the top stair , naked sword in hand , prevented , or at least endeavored to prevent , the three others from ascending . These three others fenced against him with their agile swords . D'Artagnan at first took these weapons for foils , and believed them to be buttoned ; but he soon perceived by certain scratches that every weapon was pointed and sharpened , and that at each of these scratches not only the spectators , but even the actors themselves , laughed like so many madmen . He who at the moment occupied the upper step kept his adversaries marvelously in check . A circle was formed around them . The conditions required that at every hit the man touched should quit the game , yielding his turn for the benefit of the adversary who had hit him . In five minutes three were slightly wounded , one on the hand , another on the ear , by the defen
 der of the stair , who himself remained intact--a piece of skill which was worth to him , according to the rules agreed upon , three turns of favor . However difficult it might be , or rather as he pretended it was , to astonish our young traveler , this pastime really astonished him . He had seen in his province--that land in which heads become so easily heated--a few of the preliminaries of duels ; but the daring of these four fencers appeared to him the strongest he had ever heard of even in Gascony . He believed himself transported into that famous country of giants into which Gulliver afterward went and was so frightened ; and yet he had not gained the goal , for there were still the landing place and the antechamber . On the landing they were no longer fighting , but amused themselves with stories about women , and in the antechamber , with stories about the court . On the landing d'Artagnan blushed ; in the antechamber he trembled . His warm and fickle imagination , which in 
 Gascony had rendered formidable to young chambermaids , and even sometimes their mistresses , had never dreamed , even in moments of delirium , of half the amorous wonders or a quarter of the feats of gallantry which were here set forth in connection with names the best known and with details the least concealed . But if his morals were shocked on the landing , his respect for the cardinal was scandalized in the antechamber . There , to his great astonishment , d'Artagnan heard the policy which made all Europe tremble criticized aloud and openly , as well as the private life of the cardinal , which so many great nobles had been punished for trying to pry into . That great man who was so revered by d'Artagnan the elder served as an object of ridicule to the Musketeers of Treville , who cracked their jokes upon his bandy legs and his crooked back . Some sang ballads about Mme . d'Aguillon , his mistress , and Mme . Cambalet , his niece ; while others formed parties and plans to annoy 
 the pages and guards of the cardinal duke--all things which appeared to d'Artagnan monstrous impossibilities . Nevertheless , when the name of the king was now and then uttered unthinkingly amid all these cardinal jests , a sort of gag seemed to close for a moment on all these jeering mouths . They looked hesitatingly around them , and appeared to doubt the thickness of the partition between them and the office of M. de Treville ; but a fresh allusion soon brought back the conversation to his Eminence , and then the laughter recovered its loudness and the light was not withheld from any of his actions . " Certes , these fellows will all either be imprisoned or hanged , " thought the terrified d'Artagnan , " and I , no doubt , with them ; for from the moment I have either listened to or heard them , I shall be held as an accomplice . What would my good father say , who so strongly pointed out to me the respect due to the cardinal , if he knew I was in the society of such pagans ? " W
 e have no need , therefore , to say that d'Artagnan dared not join in the conversation , only he looked with all his eyes and listened with all his ears , stretching his five senses so as to lose nothing ; and despite his confidence on the paternal admonitions , he felt himself carried by his tastes and led by his instincts to praise rather than to blame the unheard-of things which were taking place . Although he was a perfect stranger in the court of M. de Treville 's courtiers , and this his first appearance in that place , he was at length noticed , and somebody came and asked him what he wanted . At this demand d'Artagnan gave his name very modestly , emphasized the title of compatriot , and begged the servant who had put the question to him to request a moment 's audience of M. de Treville--a request which the other , with an air of protection , promised to transmit in due season . D'Artagnan , a little recovered from his first surprise , had now leisure to study costumes and p
 hysiognomy . The center of the most animated group was a Musketeer of great height and haughty countenance , dressed in a costume so peculiar as to attract general attention . He did not wear the uniform cloak--which was not obligatory at that epoch of less liberty but more independence--but a cerulean-blue doublet , a little faded and worn , and over this a magnificent baldric , worked in gold , which shone like water ripples in the sun . A long cloak of crimson velvet fell in graceful folds from his shoulders , disclosing in front the splendid baldric , from which was suspended a gigantic rapier . This Musketeer had just come off guard , complained of having a cold , and coughed from time to time affectedly . It was for this reason , as he said to those around him , that he had put on his cloak ; and while he spoke with a lofty air and twisted his mustache disdainfully , all admired his embroidered baldric , and d'Artagnan more than anyone . " What would you have ? " said the Musk
 eteer . " This fashion is coming in . It is a folly , I admit , but still it is the fashion . Besides , one must lay out one 's inheritance somehow . " " Ah , Porthos ! " cried one of his companions , " do n't try to make us believe you obtained that baldric by paternal generosity . It was given to you by that veiled lady I met you with the other Sunday , near the gate St. Honor . " " No , upon honor and by the faith of a gentleman , I bought it with the contents of my own purse , " answered he whom they designated by the name Porthos . " Yes ; about in the same manner , " said another Musketeer , " that I bought this new purse with what my mistress put into the old one . " " It 's true , though , " said Porthos ; " and the proof is that I paid twelve pistoles for it . " The wonder was increased , though the doubt continued to exist . " Is it not true , Aramis ? " said Porthos , turning toward another Musketeer . This other Musketeer formed a perfect contrast to his interrogator , w
 ho had just designated him by the name of Aramis . He was a stout man , of about two- or three-and-twenty , with an open , ingenuous countenance , a black , mild eye , and cheeks rosy and downy as an autumn peach . His delicate mustache marked a perfectly straight line upon his upper lip ; he appeared to dread to lower his hands lest their veins should swell , and he pinched the tips of his ears from time to time to preserve their delicate pink transparency . Habitually he spoke little and slowly , bowed frequently , laughed without noise , showing his teeth , which were fine and of which , as the rest of his person , he appeared to take great care . He answered the appeal of his friend by an affirmative nod of the head . This affirmation appeared to dispel all doubts with regard to the baldric . They continued to admire it , but said no more about it ; and with a rapid change of thought , the conversation passed suddenly to another subject . " What do you think of the story Chalais
  's esquire relates ? " asked another Musketeer , without addressing anyone in particular , but on the contrary speaking to everybody . " And what does he say ? " asked Porthos , in a self-sufficient tone . " He relates that he met at Brussels Rochefort , the AME DAMNEE of the cardinal disguised as a Capuchin , and that this cursed Rochefort , thanks to his disguise , had tricked Monsieur de Laigues , like a ninny as he is . " " A ninny , indeed ! " said Porthos ; " but is the matter certain ? " " I had it from Aramis , " replied the Musketeer . " Indeed ? " " Why , you knew it , Porthos , " said Aramis . " I told you of it yesterday . Let us say no more about it . " " Say no more about it ? That 's YOUR opinion ! " replied Porthos . " Say no more about it ! PESTE ! You come to your conclusions quickly . What ! The cardinal sets a spy upon a gentleman , has his letters stolen from him by means of a traitor , a brigand , a rascal-has , with the help of this spy and thanks to this cor
 respondence , Chalais 's throat cut , under the stupid pretext that he wanted to kill the king and marry Monsieur to the queen ! Nobody knew a word of this enigma . You unraveled it yesterday to the great satisfaction of all ; and while we are still gaping with wonder at the news , you come and tell us today , 'Let us say no more about it . ' " " Well , then , let us talk about it , since you desire it , " replied Aramis , patiently . " This Rochefort , " cried Porthos , " if I were the esquire of poor Chalais , should pass a minute or two very uncomfortably with me . " " And you--you would pass rather a sad quarter-hour with the Red Duke , " replied Aramis . " Oh , the Red Duke ! Bravo ! Bravo ! The Red Duke ! " cried Porthos , clapping his hands and nodding his head . " The Red Duke is capital . I 'll circulate that saying , be assured , my dear fellow . Who says this Aramis is not a wit ? What a misfortune it is you did not follow your first vocation ; what a delicious abbe you w
 ould have made ! " " Oh , it 's only a temporary postponement , " replied Aramis ; " I shall be one someday . You very well know , Porthos , that I continue to study theology for that purpose . " " He will be one , as he says , " cried Porthos ; " he will be one , sooner or later . " " Sooner . " said Aramis . " He only waits for one thing to determine him to resume his cassock , which hangs behind his uniform , " said another Musketeer . " What is he waiting for ? " asked another . " Only till the queen has given an heir to the crown of France . " " No jesting upon that subject , gentlemen , " said Porthos ; " thank God the queen is still of an age to give one ! " " They say that Monsieur de Buckingham is in France , " replied Aramis , with a significant smile which gave to this sentence , apparently so simple , a tolerably scandalous meaning . " Aramis , my good friend , this time you are wrong , " interrupted Porthos . " Your wit is always leading you beyond bounds ; if Monsieur 
 de Treville heard you , you would repent of speaking thus . " " Are you going to give me a lesson , Porthos ? " cried Aramis , from whose usually mild eye a flash passed like lightning . " My dear fellow , be a Musketeer or an abbe . Be one or the other , but not both , " replied Porthos . " You know what Athos told you the other day ; you eat at everybody 's mess . Ah , do n't be angry , I beg of you , that would be useless ; you know what is agreed upon between you , Athos and me . You go to Madame d'Aguillon 's , and you pay your court to her ; you go to Madame de Bois-Tracy 's , the cousin of Madame de Chevreuse , and you pass for being far advanced in the good graces of that lady . Oh , good Lord ! Do n't trouble yourself to reveal your good luck ; no one asks for your secret-all the world knows your discretion . But since you possess that virtue , why the devil do n't you make use of it with respect to her Majesty ? Let whoever likes talk of the king and the cardinal , and how
  he likes ; but the queen is sacred , and if anyone speaks of her , let it be respectfully . " " Porthos , you are as vain as Narcissus ; I plainly tell you so , " replied Aramis . " You know I hate moralizing , except when it is done by Athos . As to you , good sir , you wear too magnificent a baldric to be strong on that head . I will be an abbe if it suits me . In the meanwhile I am a Musketeer ; in that quality I say what I please , and at this moment it pleases me to say that you weary me . " " Aramis ! " " Porthos ! " " Gentlemen ! Gentlemen ! " cried the surrounding group . " Monsieur de Treville awaits Monsieur d'Artagnan , " cried a servant , throwing open the door of the cabinet . At this announcement , during which the door remained open , everyone became mute , and amid the general silence the young man crossed part of the length of the antechamber , and entered the apartment of the captain of the Musketeers , congratulating himself with all his heart at having so narrow
 ly escaped the end of this strange quarrel . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/56FictFlaubertG_Bovary_I5_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/56FictFlaubertG_Bovary_I5_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/56FictFlaubertG_Bovary_I5_EN.txt.txt
new file mode 100644
index 0000000..8b40599
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/56FictFlaubertG_Bovary_I5_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff I Chapter Five The brick front was just in a line with the street , or rather the road . Behind the door hung a cloak with a small collar , a bridle , and a black leather cap , and on the floor , in a corner , were a pair of leggings , still covered with dry mud . On the right was the one apartment , that was both dining and sitting room . A canary yellow paper , relieved at the top by a garland of pale flowers , was puckered everywhere over the badly stretched canvas ; white calico curtains with a red border hung crossways at the length of the window ; and on the narrow mantelpiece a clock with a head of Hippocrates shone resplendent between two plate candlesticks under oval shades . On the other side of the passage was Charles 's consulting room , a little room about six paces wide , with a table , three chairs , and an office chair . Volumes of the " Dictionary of Medical Science , " uncut , but the binding rather the worse for the successive sales through which they had gone
  , occupied almost along the six shelves of a deal bookcase . The smell of melted butter penetrated through the walls when he saw patients , just as in the kitchen one could hear the people coughing in the consulting room and recounting their histories . Then , opening on the yard , where the stable was , came a large dilapidated room with a stove , now used as a wood-house , cellar , and pantry , full of old rubbish , of empty casks , agricultural implements past service , and a mass of dusty things whose use it was impossible to guess . The garden , longer than wide , ran between two mud walls with espaliered apricots , to a hawthorn hedge that separated it from the field . In the middle was a slate sundial on a brick pedestal ; four flower beds with eglantines surrounded symmetrically the more useful kitchen garden bed . Right at the bottom , under the spruce bushes , was a cure in plaster reading his breviary . Emma went upstairs . The first room was not furnished , but in the s
 econd , which was their bedroom , was a mahogany bedstead in an alcove with red drapery . A shell box adorned the chest of drawers , and on the secretary near the window a bouquet of orange blossoms tied with white satin ribbons stood in a bottle . It was a bride 's bouquet ; it was the other one 's . She looked at it . Charles noticed it ; he took it and carried it up to the attic , while Emma seated in an arm-chair ( they were putting her things down around her ) thought of her bridal flowers packed up in a bandbox , and wondered , dreaming , what would be done with them if she were to die . During the first days she occupied herself in thinking about changes in the house . She took the shades off the candlesticks , had new wallpaper put up , the staircase repainted , and seats made in the garden round the sundial ; she even inquired how she could get a basin with a jet fountain and fishes . Finally her husband , knowing that she liked to drive out , picked up a second-hand dogcar
 t , which , with new lamps and splashboard in striped leather , looked almost like a tilbury . He was happy then , and without a care in the world . A meal together , a walk in the evening on the highroad , a gesture of her hands over her hair , the sight of her straw hat hanging from the window-fastener , and many another thing in which Charles had never dreamed of pleasure , now made up the endless round of his happiness . In bed , in the morning , by her side , on the pillow , he watched the sunlight sinking into the down on her fair cheek , half hidden by the lappets of her night-cap . Seen thus closely , her eyes looked to him enlarged , especially when , on waking up , she opened and shut them rapidly many times . Black in the shade , dark blue in broad daylight , they had , as it were , depths of different colours , that , darker in the centre , grew paler towards the surface of the eye . His own eyes lost themselves in these depths ; he saw himself in miniature down to the s
 houlders , with his handkerchief round his head and the top of his shirt open . He rose . She came to the window to see him off , and stayed leaning on the sill between two pots of geranium , clad in her dressing gown hanging loosely about her . Charles , in the street buckled his spurs , his foot on the mounting stone , while she talked to him from above , picking with her mouth some scrap of flower or leaf that she blew out at him . Then this , eddying , floating , described semicircles in the air like a bird , and was caught before it reached the ground in the ill-groomed mane of the old white mare standing motionless at the door . Charles from horseback threw her a kiss ; she answered with a nod ; she shut the window , and he set off . And then along the highroad , spreading out its long ribbon of dust , along the deep lanes that the trees bent over as in arbours , along paths where the corn reached to the knees , with the sun on his back and the morning air in his nostrils , hi
 s heart full of the joys of the past night , his mind at rest , his flesh at ease , he went on , re-chewing his happiness , like those who after dinner taste again the truffles which they are digesting . Until now what good had he had of his life ? His time at school , when he remained shut up within the high walls , alone , in the midst of companions richer than he or cleverer at their work , who laughed at his accent , who jeered at his clothes , and whose mothers came to the school with cakes in their muffs ? Later on , when he studied medicine , and never had his purse full enough to treat some little work-girl who would have become his mistress ? Afterwards , he had lived fourteen months with the widow , whose feet in bed were cold as icicles . But now he had for life this beautiful woman whom he adored . For him the universe did not extend beyond the circumference of her petticoat , and he reproached himself with not loving her . He wanted to see her again ; he turned back qui
 ckly , ran up the stairs with a beating heart . Emma , in her room , was dressing ; he came up on tiptoe , kissed her back ; she gave a cry . He could not keep from constantly touching her comb , her ring , her fichu ; sometimes he gave her great sounding kisses with all his mouth on her cheeks , or else little kisses in a row all along her bare arm from the tip of her fingers up to her shoulder , and she put him away half-smiling , half-vexed , as you do a child who hangs about you . Before marriage she thought herself in love ; but the happiness that should have followed this love not having come , she must , she thought , have been mistaken . And Emma tried to find out what one meant exactly in life by the words felicity , passion , rapture , that had seemed to her so beautiful in books . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/57FictFlaubertG_Bovary_III3_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/57FictFlaubertG_Bovary_III3_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/57FictFlaubertG_Bovary_III3_EN.txt.txt
new file mode 100644
index 0000000..520c389
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/57FictFlaubertG_Bovary_III3_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff III Chapter Three They were three full , exquisite days--a true honeymoon . They were at the Hotel-de-Boulogne , on the harbour ; and they lived there , with drawn blinds and closed doors , with flowers on the floor , and iced syrups were brought them early in the morning . Towards evening they took a covered boat and went to dine on one of the islands . It was the time when one hears by the side of the dockyard the caulking-mallets sounding against the hull of vessels . The smoke of the tar rose up between the trees ; there were large fatty drops on the water , undulating in the purple colour of the sun , like floating plaques of Florentine bronze . They rowed down in the midst of moored boats , whose long oblique cables grazed lightly against the bottom of the boat . The din of the town gradually grew distant ; the rolling of carriages , the tumult of voices , the yelping of dogs on the decks of vessels . She took off her bonnet , and they landed on their island . They sat dow
 n in the low-ceilinged room of a tavern , at whose door hung black nets . They ate fried smelts , cream and cherries . They lay down upon the grass ; they kissed behind the poplars ; and they would fain , like two Robinsons , have lived for ever in this little place , which seemed to them in their beatitude the most magnificent on earth . It was not the first time that they had seen trees , a blue sky , meadows ; that they had heard the water flowing and the wind blowing in the leaves ; but , no doubt , they had never admired all this , as if Nature had not existed before , or had only begun to be beautiful since the gratification of their desires . At night they returned . The boat glided along the shores of the islands . They sat at the bottom , both hidden by the shade , in silence . The square oars rang in the iron thwarts , and , in the stillness , seemed to mark time , like the beating of a metronome , while at the stern the rudder that trailed behind never ceased its gentle s
 plash against the water . Once the moon rose ; they did not fail to make fine phrases , finding the orb melancholy and full of poetry . She even began to sing-- " One night , do you remember , we were sailing , " etc. Her musical but weak voice died away along the waves , and the winds carried off the trills that Leon heard pass like the flapping of wings about him . She was opposite him , leaning against the partition of the shallop , through one of whose raised blinds the moon streamed in . Her black dress , whose drapery spread out like a fan , made her seem more slender , taller . Her head was raised , her hands clasped , her eyes turned towards heaven . At times the shadow of the willows hid her completely ; then she reappeared suddenly , like a vision in the moonlight . Leon , on the floor by her side , found under his hand a ribbon of scarlet silk . The boatman looked at it , and at last said-- " Perhaps it belongs to the party I took out the other day . A lot of jolly folk ,
  gentlemen and ladies , with cakes , champagne , cornets--everything in style ! There was one especially , a tall handsome man with small moustaches , who was that funny ! And they all kept saying , 'Now tell us something , Adolphe--Dolpe , ' I think . " She shivered . " You are in pain ? " asked Leon , coming closer to her . " Oh , it 's nothing ! No doubt , it is only the night air . " " And who does n't want for women , either , " softly added the sailor , thinking he was paying the stranger a compliment . Then , spitting on his hands , he took the oars again . Yet they had to part . The adieux were sad . He was to send his letters to Mere Rollet , and she gave him such precise instructions about a double envelope that he admired greatly her amorous astuteness . " So you can assure me it is all right ? " she said with her last kiss . " Yes , certainly . " " But why , " he thought afterwards as he came back through the streets alone , " is she so very anxious to get this power of 
 attorney ? " 
\ No newline at end of file


[09/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/165ted_michael_sandel_the_lost_art_of_democratic_debate.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/165ted_michael_sandel_the_lost_art_of_democratic_debate.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/165ted_michael_sandel_the_lost_art_of_democratic_debate.txt
new file mode 100644
index 0000000..5b80dc4
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/165ted_michael_sandel_the_lost_art_of_democratic_debate.txt
@@ -0,0 +1,2 @@
+
+One thing the world needs , one thing this country desperately needs is a better way of conducting our political debates . We need to rediscover the lost art of democratic argument . ( Applause ) If you think about the arguments we have , most of the time it 's shouting matches on cable television , ideological food fights on the floor of Congress . I have a suggestion . Look at all the arguments we have these days over health care , over bonuses and bailouts on Wall Street , over the gap between rich and poor , over affirmative action and same-sex marriage . Lying just beneath the surface of those arguments , with passions raging on all sides , are big questions of moral philosophy , big questions of justice . But we too rarely articulate and defend and argue about those big moral questions in our politics . So what I would like to do today is have something of a discussion . First , let me take a famous philosopher who wrote about those questions of justice and morality , give you
  a very short lecture on Aristotle of ancient Athens , Aristotle 's theory of justice , and then have a discussion here to see whether Aristotle 's ideas actually inform the way we think and argue about questions today . So , are you ready for the lecture ? According to Aristotle justice means giving people what they deserve . That 's it ; that 's the lecture . ( Laughter ) Now , you may say , well , that 's obvious enough . The real questions begin when it comes to arguing about who deserves what and why . Take the example of flutes . Suppose we 're distributing flutes . Who should get the best ones ? Let 's see what people -- What would you say ? Who should get the best flute ? You can just call it out . ( Audience : Random . ) Michael Sandel : At random . You would do it by lottery . Or by the first person to rush into the hall to get them . Who else ? ( Audience : The best flute players . ) MS : The best flute players . ( Audience : The worst flute players . ) MS : The worst flu
 te players . How many say the best flute players ? Why ? Actually , that was Aristotle 's answer too . ( Laughter ) But here 's a harder question . Why do you think , those of you who voted this way , that the best flutes should go to the best flute players ? Peter : The greatest benefit to all . MS : The greatest benefit to all . We 'll hear better music if the best flutes should go to the best flute players . That 's Peter ? ( Audience : Peter . ) MS : All right . Well , it 's a good reason . We 'll all be better off if good music is played rather than terrible music . But Peter , Aristotle does n't agree with you that that 's the reason . That 's all right . Aristotle had a different reason for saying the best flutes should go to the best flute players . He said , that 's what flutes are for -- to be played well . He says that to reason about just distribution of a thing , we have to reason about , and sometimes argue about , the purpose of the thing , or the social activity , in
  this case , musical performance . And the point , the essential nature , of musical performance is to produce excellent music . It 'll be a happy byproduct that we 'll all benefit . But when we think about justice , Aristotle says , what we really need to think about is the essential nature of the activity in question and the qualities that are worth honoring and admiring and recognizing . One of the reasons that the best flute players should get the best flutes is that musical performance is not only to make the rest of us happy , but to honor and recognize the excellence of the best musicians . Now , flutes may seem ... the distribution of flutes may seem a trivial case . Let 's take a contemporary example of the dispute about justice . It had to do with golf . Casey Martin -- a few years ago , Casey Martin -- did any of you hear about him ? He was a very good golfer , but he had a disability . he had a bad leg , a circulatory problem , that made it very painful for him to walk t
 he course . In fact , it carried risk of injury . He asked the PGA , the Professional Golfers ' Association , for permission to use a golf cart in the PGA tournaments . They said , " No. Now that would give you an unfair advantage . " He sued , and his case went all the way to the Supreme Court , believe it or not , the case over the golf cart . Because the law says that the disabled must be accommodated , provided the accommodation does not change the essential nature of the activity . He says , " I 'm a great golfer . I want to compete . But I need a golf cart to get from one hole to the next . " Suppose you were on the Supreme Court . Suppose you were deciding the justice of this case . How many here would say that Casey Martin does have a right to use a golf cart ? And how many say , no , he does n't ? All right , let 's take a poll , show of hands . How many would rule in favor of Casey Martin ? And how many would not ? How many would say he does n't ? All right , we have a goo
 d division of opinion here . Someone who would not grant Casey Martin the right to a golf cart , what would be your reason ? Raise your hand , and we 'll try to get you a microphone . What would be your reason ? ( Audience : It 'd be an unfair advantage . ) MS : It would be an unfair advantage if he gets to ride in a golf cart . All right , those of you , I imagine most of you who would not give him the golf cart worry about an unfair advantage . What about those of you who say he should be given a golf cart ? How would you answer the objection ? Yes , all right . Audience : The cart 's not part of the game . MS : What 's your name ? ( Audience : Charlie . ) MS : Charlie says -- We 'll get Charlie a microphone in case someone wants to reply . Tell us , Charlie , why would you say he should be able to use a golf cart ? Charlie : The cart 's not part of the game . MS : But what about walking from hole to hole ? Charlie : It does n't matter ; it 's not part of the game . MS : Walking t
 he course is not part of the game of golf ? Charlie : Not in my book , it is n't . MS : All right . Stay there , Charlie . ( Laughter ) Who has an answer for Charlie ? All right , who has an answer for Charlie ? What would you say ? Audience : The endurance element is a very important part of the game , walking all those holes . MS : Walking all those holes ? That 's part of the game of golf ? ( Audience : Absolutely . ) MS : What 's your name ? ( Audience : Warren . ) MS : Warren . Charlie , what do you say to Warren ? Charley : I 'll stick to my original thesis . ( Laughter ) MS : Warren , are you a golfer ? Warren : I am not a golfer . Charley : And I am . ( MS : Okay . ) ( Laughter ) ( Applause ) You know , it 's interesting . In the case , in the lower court , they brought in golfing greats to testify on this very issue . Is walking the course essential to the game ? And they brought in Jack Nicklaus and Arnold Palmer . And what do you suppose they all said ? Yes . They agreed 
 with Warren . They said , yes , walking the course is strenuous physical exercise . The fatigue factor is an important part of golf . And so it would change the fundamental nature of the game to give him the golf cart . Now , notice , something interesting -- Well , I should tell you about the Supreme Court first . The Supreme Court decided . What do you suppose they said ? They said yes , that Casey Martin must be provided a golf cart . Seven to two , they ruled . What was interesting about their ruling and about the discussion we 've just had is that the discussion about the right , the justice , of the matter depended on figuring out what is the essential nature of golf . And the Supreme Court justices wrestled with that question . And Justice Stevens , writing for the majority , said he had read all about the history of golf , and the essential point of the game is to get very small ball from one place into a hole in as few strokes as possible , and that walking was not essentia
 l , but incidental . Now , there were two dissenters , one of whom was Justice Scalia . He would n't have granted the cart , and he had a very interesting dissent . It 's interesting because he rejected the Aristotelian premise underlying the majority 's opinion . He said it 's not possible to determine the essential nature of a game like golf . Here 's how he put it . " To say that something is essential is ordinarily to say that it is necessary to the achievement of a certain object . But since it is the very nature of a game to have no object except amusement , ( Laughter ) that is , what distinguishes games from productive activity , ( Laughter ) it is quite impossible to say that any of a game 's arbitrary rules is essential . " So there you have Justice Scalia taking on the Aristotelian premise of the majority 's opinion . Justice Scalia 's opinion is questionable for two reasons . First , no real sports fan would talk that way . ( Laughter ) If we had thought that the rules o
 f the sports we care about are merely arbitrary , rather than designed to call forth the virtues and the excellences that we think are worthy of admiring , we would n't care about the outcome of the game . It 's also objectionable on a second ground . On the face of it , it seemed to be -- this debate about the golf cart -- an argument about fairness , what 's an unfair advantage . But if fairness were the only thing at stake , there would have been an easy and obvious solution . What would it be ? ( Audience : Let everyone use the cart . ) Let everyone ride in a golf cart if they want to . Then the fairness objection goes away . But letting everyone ride in a cart would have been , I suspect , more anathema to the golfing greats and to the PGA , even than making an exception for Casey Martin . Why ? Because what was at stake in the dispute over the golf cart was not only the essential nature of golf , but , relatedly , the question , what abilities are worthy of honor and recogniti
 on as athletic talents ? Let me put the point as delicately as possible : Golfers are a little sensitive about the athletic status of their game . ( Laughter ) After all , there 's no running or jumping , and the ball stands still . ( Laughter ) So if golfing is the kind of game that can be played while riding around in a golf cart , it would be hard to confer on the golfing greats the status that we confer , the honor and recognition that goes to truly great athletes . That illustrates that with golf , as with flutes , it 's hard to decide the question of what justice requires , without grappling with the question " What is the essential nature of the activity in question , and what qualities , what excellences connected with that activity , are worthy of honor and recognition ? " Let 's take a final example that 's prominent in contemporary political debate : same-sex marriage . There are those who favor state recognition only of traditional marriage between one man and one woman 
 , and there are those who favor state recognition of same-sex marriage . How many here favor the first policy : the state should recognize traditional marriage only ? And how many favor the second , same-sex marriage ? Now , put it this way , what ways of thinking about justice and morality underlie the arguments we have over marriage ? The opponents of same-sex marriage say that the purpose of marriage , fundamentally , is procreation , and that 's what 's worthy of honoring and recognizing and encouraging . And the defenders of same-sex marriage say no , procreation is not the only purpose of marriage . What about a lifelong , mutual , loving commitment ? That 's really what marriage is about . So with flutes , with golf carts , and even with a fiercely contested question like same-sex marriage , Aristotle has a point . Very hard to argue about justice without first arguing about the purpose of social institutions and about what qualities are worthy of honor and recognition . So l
 et 's step back from these cases and see how they shed light on the way we might improve , elevate , the terms of political discourse in the United States , and for that matter , around the world . There is a tendency to think that if we engage too directly with moral questions in politics , that 's a recipe for disagreement , and for that matter , a recipe for intolerance and coercion . So better to shy away from , to ignore , the moral and the religious convictions that people bring to civic life . It seems to me that our discussion reflects the opposite , that a better way to mutual respect is to engage directly with the moral convictions citizens bring to public life , rather than to require that people leave their deepest moral convictions outside politics before they enter . That , it seems to me , is a way to begin to restore the art of democratic argument . Thank you very much . ( Applause ) Thank you . ( Applause ) Thank you . ( Applause ) Thank you very much . Thanks . Tha
 nk you . Chris . Thanks , Chris . Chris Anderson : From flutes to golf courses to same-sex marriage . That was a genius link . Now look , you 're a pioneer of open education . Your lecture series was one of the first to do it big . What 's your vision for the next phase of this ? MS : Well , I think that it is possible . In the classroom , we have arguments on some of the most fiercely held moral convictions that students have about big public questions . And I think we can do that in public life more generally . And so my real dream would be to take the public television series that we 've created of the course -- it 's available now , online , free for everyone anywhere in the world -- and to see whether we can partner with institutions , at universities in China , in India , in Africa , around the world , to try to promote civic education and also a richer kind of democratic debate . CA : So you picture , at some point , live , in real time , you could have this kind of conversat
 ion , inviting questions , but with people from China and India joining in ? MS : Right . We did a little bit of it here with 1,500 people in Long Beach , and we do it in a classroom at Harvard with about 1,000 students . Would n't it be interesting to take this way of thinking and arguing , engaging seriously with big moral questions , exploring cultural differences and connect through a live video hookup , students in Beijing and Mumbai and in Cambridge , Massachusetts and create a global classroom . That 's what I would love to do . ( Applause ) CA : So , I would imagine that there are a lot of people who would love to join you in that endeavor . Michael Sandel . Thank you so much . ( MS : Thanks so much . ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/166ted_michael_shermer_on_believing_strange_things.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/166ted_michael_shermer_on_believing_strange_things.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/166ted_michael_shermer_on_believing_strange_things.txt
new file mode 100644
index 0000000..e93d7fe
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/166ted_michael_shermer_on_believing_strange_things.txt
@@ -0,0 +1,2 @@
+
+Hey , I am Michael Shermer , the director of the Skeptics Society , the publisher of Skeptic magazine . We investigate claims of the paranormal , pseudo-science , and fringe groups and cults and claims of all kinds between -- science and pseudo-science and non-science and junk science , voodoo science , pathological science , bad science , non-science and plain old nonsense . And unless you 've been on Mars recently , you know there 's a lot of that out there . Some people call us debunkers , which is kind of a negative term . But let 's face it -- there 's a lot of bunk , and we are like the bunko squads of the police departments out there , flushing out . Well , we 're sort of like the Ralph Naders of bad ideas -- ( Laughter ) -- trying to replace bad ideas with good ideas . I 'll show you an example of a bad idea . I brought this with me . This was given to us by NBC Dateline to test . It 's the -- it 's produced by the Quadro Corporation of West Virginia . It 's called the Quadr
 o 2000 Dowser Rod . ( Laughter ) This was being sold to high school administrators for 900 dollars a piece . It 's a piece of plastic with a Radio Shack antenna attached to it . You could dowse for all sorts of things , but this particular one was built to dowse for marijuana in students ' lockers . ( Laughter ) So the way it works is , you go down the hallway and you see if it tilts toward a particular locker , and then you open the locker . So it looks something like this . I 'll show you . ( Laughter ) No , it -- well , it has kind of a right-leaning bias . So , I 'll show -- well , this is science , so we 'll do a controlled experiment . It 'll go this way for sure . ( Laughter ) Sir , you want to empty your pockets . Please , sir ? ( Laughter ) So the question was , can it actually find marijuana in students ' lockers ? And the answer is , If you open enough of them -- yes . ( Laughter ) ( Applause ) But in science , we have to keep track of the misses , not just the hits . And
  that 's probably the key lesson to my short talk here , is that this is how psychics work , astrologers , and tarot card readers and so on . People remember the hits ; they forget the misses . In science we have to keep the whole database , and look to see if the number of hits is somehow stands out from the total number that you would expect by chance . In this case , we tested it . We had two opaque boxes , one with government-approved THC marijuana , and one with nothing . And it got it 50 percent of the time -- -- which is exactly what you 'd expect with a coin flip model . So that 's just a fun little example here of the sorts of things we do . Skeptic is the quarterly publication . Each one has a particular theme , like this one is on the future of intelligence . Are people getting smarter or dumber ? I have an opinion of this myself because the business I 'm in . But , in fact , people , it turns out , are getting smarter . Three IQ points per 10 years , going up . Sort of a
 n interesting thing . With science , do n't think of skepticism as a thing or even science as a thing . Are science and religion compatible ? It 's like , are science and plumbing compatible ? These -- they 're just two different things . Science is not a thing . It 's a verb . It 's a way of thinking about things . It 's a way of looking for natural explanations for all phenomena . I mean , what 's more likely -- that extraterrestrial intelligences or multi-dimensional beings travel across the vast distances of interstellar space to leave a crop circle in Farmer Bob 's field in Puckerbrush , Kansas to promote skeptic . com , our webpage ? Or is it more likely that a reader of Skeptic did this with Photoshop ? And in all cases we have to ask -- ( Laughter ) -- what 's the more likely explanation ? And before we say something is out of this world , we should first make sure that it 's not in this world . What 's more likely -- that Arnold had a little extraterrestrial help in his run
  for the governorship ? Or that the World Weekly News makes stuff up ? ( Laughter ) And part of that -- the same theme is expressed nicely here in this Sidney Harris cartoon . For those of you in the back , it says here " Then a miracle occurs . I think you need to be more explicit here in step two . " This single slide completely dismantles the intelligent design arguments . There 's nothing more to it than that . ( Applause ) You can say a miracle occurs . It 's just that it does n't explain anything . It does n't offer anything . There 's nothing to test . It 's the end of the conversation for intelligent design creationists . Whereas -- and it 's true , scientists sometimes throw terms out as linguistic place fillers -- dark energy or dark matter or something like that . Until we figure out what it is , we 'll just call it this . It 's the beginning of the causal chain for science . For intelligent design creationists , it 's the end of the chain . So again , we can ask this -- 
 what 's more likely -- are UFOs alien spaceships or perceptual cognitive mistakes , or even fakes ? This is a UFO shot from my house in Altadena , California , looking down over Pasadena . And if it looks a lot like a Buick hubcap , it 's because it is . You do n't even need Photoshop , you do n't need high-tech equipment , you do n't need computers . This was shot with a throw-away Kodak Instamatic camera . You just have somebody off on the side with a hubcap ready to go . Camera 's ready -- that 's it . ( Laughter ) So , although it 's possible that most of these things are fake or illusions or so on and that some of them are real , it 's more likely that all of them are fake , like the crop circles . On a more serious note , in all of science we 're looking for a balance between data and theory . In the case of Galileo , he had two problems when he turned his telescope to Saturn . First of all , there was no theory of planetary rings . And second of all , his data was grainy and 
 fuzzy , and he could n't quite make out what it was he was looking at . So he wrote that he had seen -- " I have observed that the furthest planet has three bodies . " And this is what he ended up concluding that he saw . So without a theory of planetary rings and with only grainy data , you ca n't have a good theory . And it was n't solved until 1655. This is Christiaan Huygens 's book in which he cataloged all the mistakes that people made in trying to figure out what was going on with Saturn . It was n't till -- Huygens had two things . He had a good theory of planetary rings and how the solar system operated . And then , he had better telescopic , more fine-grain data in which he could figure out that as the Earth is going around faster -- according to Kepler 's Laws -- than Saturn , then we catch up with it . And we see the angles of the rings at different angles , there . And that , in fact , turns out to be true . The problems with having a theory is that your theory may be l
 oaded with cognitive biases . So one of the problems of explaining why people believe weird things is that we have things on a simple level . And then I 'll go to more serious ones . Like , we have a tendency to see faces . This is the face on Mars which was -- in 1976 , where there was a whole movement to get NASA to photograph that area because people thought this was monumental architecture made by Martians . Well , it turns out -- here 's the close-up of it from 2001. If you squint , you can still see the face . And when you 're squinting , what you 're doing is you 're turning that from fine-grain to coarse-grain . And so , you 're reducing the quality of your data . And if I did n't tell you what to look for , you 'd still see the face , because we 're programmed by evolution to see faces . Faces are important for us socially . And , of course , happy faces . Faces of all kinds are easy to see . ( Laughter ) You can see the happy face on Mars , there . If astronomers were frog
 s perhaps they 'd see Kermit the Frog . Do you see him there ? Little froggy legs . Or if geologists were elephants ? Religious iconography . ( Laughter ) Discovered by a Tennessee baker in 1996. He charged five bucks a head to come see the nun bun till he got a cease-and-desist from Mother Teresa 's lawyer . Here 's Our Lady of Guadalupe and Our Lady of Watsonville , just down the street . Or is it up the street from here ? Tree bark is particularly good because it 's nice and grainy , branchy , black-and-white splotchy and you can get the pattern-seeking -- humans are pattern-seeking animals . Here 's the Virgin Mary on the side of a glass window in Sao Paulo . Now , here 's the Virgin Mary made her appearance on a cheese sandwich -- which I got to actually hold in a Las Vegas casino , of course , this being America . ( Laughter ) This casino paid 28,500 dollars on eBay for the cheese sandwich . ( Laughter ) But who does it really look like , the Virgin Mary ? ( Laughter ) It has 
 that sort of puckered lips , 1940s-era look . Virgin Mary in Clearwater , Florida . I actually went to see this one . There was a lot of people there -- the faithful come to be in their -- wheelchairs and crutches , and so on . And we went down , investigated . Just to give you a size -- that 's Dawkins , me and The Amazing Randi , next to this two , two and a half story size image . All these candles , so many thousands of candles people had lit in tribute to this . So we walked around the backside , just to see what was going on here , where it turns out wherever there 's a sprinkler head and a palm tree , you get the effect . Here 's the Virgin Mary on the backside , which they started to wipe off . I guess you can only have one miracle per building . ( Laughter ) So is it really a miracle of Mary , or is it a miracle of Marge ? ( Laughter ) And then I 'm going to finish up with another example of this with audio -- auditory illusions . There is this film , " White Noise , " with
  Michael Keaton about the dead talking back to us . By the way , this whole business of talking to the dead , it 's not that big a deal . Anybody can do it , turns out . It 's getting the dead to talk back that 's the really hard part . ( Laughter ) In this case , supposedly , these messages are hidden in electronic phenomena . There 's a ReverseSpeech . com web page on which I downloaded this stuff . Here is the forward -- this is the most famous one of all of these . Here 's the forward version of the very famous song . Boy , coudl n't you just listen to that all day ? ( Laughter ) All right , here it is backwards , and see if you can hear the hidden messages that are supposedly in there . What did you get ? ( Audience : Satan . ) Michael Shermer : Satan ? OK , well , at least we got Satan . Now , I 'll prime your auditory part of your brain to tell you what you 're supposed to hear , and then hear it again . ( Laughter ) ( Applause ) You ca n't miss it when I tell you what 's the
 re . ( Laughter ) All right , I 'm going to just end with a positive , nice , little story about -- the Skeptics is a nonprofit educational organization . We 're always looking for little , good things that people do . And in England , there 's a pop singer . Very -- one of the top popular singers in England today , Katie Melua . And she wrote a beautiful song . It was in the top five in 2005 , called , " Nine Million Bicycles in Beijing . " It 's a love story -- she 's sort of the Norah Jones of the U. K. -- about how she much loves her guy , and compared to nine million bicycles , and so forth . And she has this one passage here . \u266b We are 12 billion light-years from the edge \u266b \u266b That 's a guess \u266b \u266b No one can ever say it 's true \u266b \u266b But I know that I will always be with you \u266b Well , that 's nice . At least she got it close . In America it would be , " We 're 6,000 light years from the edge . " ( Laughter ) But my friend , Simon Singh , the particle physicist , now
  turned science educator , and he wrote the book " The Big Bang , " and so on . He uses every chance he gets to promote good science . And so , he wrote an op-ed piece in The Guardian about Katie 's song , in which he said , well , we know exactly how old , how far from the edge . You know , it 's 12 -- it 's 13.7 billion light years , and it 's not a guess . We know within precise error bars there how close it is . And so , we can say , although not absolutely true , that it 's pretty close to being true . And , to his credit , Katie called him up after this op-ed piece came out . And said , " I 'm so embarrassed . I was a member of the astronomy club , and I should have known better . " And she re-cut the song . So I 'll end with the new version . \u266b We are 13.7 billion light years \u266b \u266b from the edge of the observable universe \u266b \u266b That 's a good estimate with well-defined error bars \u266b \u266b And with the available information \u266b \u266b I predict that I will always be with you
  \u266b ( Applause ) How cool is that ? ( Applause ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/167ted_misha_glenny_investigates_global_crime_networks.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/167ted_misha_glenny_investigates_global_crime_networks.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/167ted_misha_glenny_investigates_global_crime_networks.txt
new file mode 100644
index 0000000..96e6dfa
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/167ted_misha_glenny_investigates_global_crime_networks.txt
@@ -0,0 +1,2 @@
+
+These are grim economic times fellow TEDsters , grim economic times indeed . And so , I would like to cheer you up with one of the great , albeit largely unknown , commercial success stories of the past 20 years . Comparable , in its own very peculiar way , to the achievements of Microsoft or Google . And it 's an industry which has bucked the current recession with equanimity . I refer to organized crime . Now organized crime has been around for a very long time , I hear you say . And these would be wise words , indeed . But in the last two decades it has experienced an unprecedented expansion . Now accounting for roughly 15 percent of the world 's GDP . I like to call it the Global Shadow Economy , or McMafia , for short . So what triggered this extraordinary growth in cross-border crime ? Well , of course , there is globalization , technology , communications , all that stuff , which we 'll talk about a little bit later . But first , I would like to take you back to this event . 
 The collapse of communism . All across Eastern Europe , a most momentous episode in our post-war history . Now it 's time for full disclosure . This event meant a great deal to me personally . I had started smuggling books across the iron curtain to Democratic opposition groups in Eastern Europe , like Solidarity in Poland , when I was in my teens . I then started writing about Eastern Europe , and eventually I became the BBC 's chief correspondent for the region . Which is what I was doing in 1989. And so when 425 million people finally won the right to chose their own governments I was ecstatic . But I was also a touch worried about some of the nastier things lurking behind the wall . It was n't long , for example , before ethnic nationalism reared its bloody head in Yugoslavia . And amongst the chaos , amidst the euphoria , it took me a little while to understand that some of the people who had wielded power before 1989 , in Eastern Europe , continued to do so after the revolutio
 ns there . Obviously there were characters like this . But there were also more unexpected people who played a critical role in what was going on in Eastern Europe . Like this character . Remember these guys ? They used to win the gold medals in weightlifting and wrestling , every four years in the Olympics . And they were the great celebrities of communism . With a fabulous lifestyle to go with it . The used to get great apartments in the center of town . Casual sex on tap . And they could travel to the West very freely , which was a great luxury at the time . It may come as a surprise , but they played a critical role in the emergence of the market economy in Eastern Europe . Or as I like to call them , they are the midwives of capitalism . Here are some of those same wieghtlifters after their 1989 makeover . Now in Bulgaria , this photograph was taken in Bulgaria , when communism collapsed all over Eastern Europe it was n't just communism , it was the state that collapsed as well
  . That means your police force was n't working . The court system was n't functioning properly . So what was a business man in the brave new world of East European capitalism going to do to make sure that his contracts would be honored ? Well , he would turn to people who were called , rather prosaically by sociologists , privatized law enforcement agencies . We prefer to know them as the mafia . And in Bulgaria , the mafia was soon joined with 14 thousand people who were sacked from their jobs in the security services between 1989 and 1991. Now , when your state is collapsing , your economy is heading south at a rate of knots , the last people you want coming on to the labor market are 14 thousand men and women whose chief skills are surveillance , are smuggling , building underground networks , and killing people . But that 's what happened all over Eastern Europe . Now , when I was working in the 1990s I spent most of the time covering the appalling conflict in Yugoslavia . And 
 I could n't help notice that the people who were perpetrating the appalling atrocities , the paramilitary organizations , were actually the same people running the organized criminal syndicates . And I came to think that behind the violence lay a sinister criminal enterprise . And so I resolved to travel around the world examining this global criminal underworld by talking to policemen , by talking to victims , by talking to consumers of illicit goods and services . But above all else , by talking to the gangsters themselves . And the Balkans was a fabulous place to start . Why ? Well of course there was the issue of law and order collapsing . But also , as they say in the retail trade , it 's location location location . And what I noticed at the beginning of my research that the Balkans had turned into a vast transit zone for illicit goods and services coming from all over the world . Heroin , cocaine , women being trafficked into prostitution and precious minerals . And where wer
 e they heading ? The European Union , which by now was beginning to reap the benefits of globalization . Transforming it into the most affluent consumer market in history . Eventually comprising some 500 million people . And a significant minority of those 500 million people like to spend some of their leisure time and spare cash sleeping with prostitutes , sticking 50 Euro notes up their nose and employing illegal migrant laborers . Now , organized crime in a globalizing world operates in the same way as any other business . It has zones of production , like Afghanistan and Columbia . It has zones of distribution , like Mexico and the Balkans . And then , of course , it has zones of consumption , like the European Union , Japan and of course , the United States . The zones of production and distribution tend to lie in the developing world . And they are often threatened by appalling violence and bloodshed . Take Mexico for example . Six thousand people killed there in the last 18 m
 onths as a direct consequence of the cocaine trade . But what about the Democratic Republic of Congo ? Since 1998 , five million people have died there . It 's not a conflict you read about much in the newspapers . But it 's the biggest conflict on this planet since the Second World War . And why is it ? Because mafias from all around the world cooperate with local paramilitaries in order to seize the supplies of the rich mineral resources of the region . In the year 2000 , 80 percent of the world 's coltan was sourced to the killing fields of the eastern Democratic Republic of Congo . Now coltan you will find in almost every mobile phone , in almost every laptop and games console . The Cogolese war lords were selling this stuff to the mafia in exchange for weapons . And the mafia would then sell it on to Western markets . And it is this Western desire to consume that is the primary driver of international organized crime . Now , let me show you some of my friends in action , caught
  conveniently on film by the Italian police and smuggling duty-not-paid cigarettes . Now , cigarettes out the factory gate are very cheap . The European Union then imposes the highest taxes on them in the world . So if you can smuggle them into the E. U. there are very handsome profits to be made . And I want to show you this to demonstrate the type of resources available to these groups . This boat is worth one million Euros when it 's new . And it 's the fastest thing on European waters . From 1994 , for seven years , 20 of these boats made the trip across the Adriatic , from Montenegro to Italy , every single night . And as a consequence of this trade Britain alone lost eight billion dollars in revenue . And instead that money went to underwrite the wars in Yugoslavia and line the pockets of unscrupulous individuals . Now Italian police , when this trade started , had just two boats which could go at the same speed . And this is very important , because the only way you can catch
  these guys is if they run out of gas . Sometimes the gangsters would bring with them women being trafficked into prostitution . And if the police intervened , they would hurl the women into the sea so that the police had to go and save them from drowning , rather than chasing the bad guys . So I have shown you this to demonstrate how many boats , how many vessels it takes to catch one of these guys . And the answer is six vessels . And remember , 20 of these speed boats were coming across the Adriatic every single night . So what were these guys doing with all the money they were making ? Well , this is where we come to globalization because that was not just the deregulation of global trade . It was the liberalization of international financial markets . And boy did that make it easy for the money launderers . The last two decades have been the champagne era for dirty lucre . In the 1990s we saw financial centers around the world competing for their business . And there was simply
  no effective mechanism to prevent money laundering . And a lot of licit banks were also happy to accept deposits from very dubious sources without questions being asked . But at the heart of this , is the offshore banking network . Now these things are an essential part of the money laundering parade . And if you want to do something about illegal tax evasion and transnational organized crime , money laundering , you have to get rid of them . On a positive note , we at last have someone in the White House who has consistently spoken out against these corrosive entities . And if anyone is concerned about what I believe is the necessity for new legislation , regulation , effective regulation , I say , let 's take a look at Bernie Madoff , who is now going to be spending the rest of his life in jail . Bernie Madoff stole 65 billion dollars . That puts him up there on the Olympus of gangsters with the Colombian cartels , and the major Russian crime syndicates . But he did this for deca
 des in the very heart of Wall Street . And no regulator picked up on it . So how many other Madoffs are there on Wall street , or in the city of London , fleecing ordinary folk , and money laundering ? Well I can tell you , it 's quite a few of them . Let me go on to the 101 of international organized crime now . And that is narcotics . Our second marijuana farm photograph for the morning . This one , however , is in central British Columbia where I photographed it . It 's one of the tens of thousands of mom-and-pop grow-ops in B. C. which ensure that over five percent of the province 's GDP is accounted for by this trade . Now I was taken by inspector Brian Cantera , of the Royal Canadian Mounted Police , to a cavernous warehouse east of Vancouver to see some of the goods which are regularly confiscated by the RCMP from the smugglers who are sending it , of course , down south to the United States where there is an insatiable market for B. C. Bud , as it 's called , in part because
  it 's marketed as organic , which of course goes down very well in California . ( Laughter ) ( Applause ) Now even by the police 's asmission this makes not a dent in the profits , really , of the major exporters . Since the beginning of globalization the global narcotics market has expanded enormously . There has , however , been no concomitant increase in the resources available to police forces . This , however , may all be about to change . Because something very strange is going on . The United Nations recognized earlier this , it was last month actually , that Canada has become a key area of distribution and production of ecstasy and other synthetic drugs . Interestingly , the market share of heroin and cocaine is going down because the pills are getting ever better at reproducing their highs . Now that is a game changer . Because it shifts production away from the developing world and into the Western world . When that happens it is a trend which is set to overwhelm our poli
 cing capacity in the West . The drugs policy which we 've had in place for 40 years is long overdue for a very serious rethink , in my opinion . Now , the recession . Well , organized crime has already adapted very well to the recession . Not surprising , the most opportunistic industry in the whole world . And it has no rules to its regulatory system . Except , of course , it has two business risks , arrest by law enforcement , which is , frankly , the least of their worries , and competition from other groups , i. e. a bullet in the back of the head . What they 've done is they 've shifted their operations . People do n't smoke as much dope , or visit prostitutes quite so frequently during a recession . And so instead they have invaded financial and corporate crime in a big way , but above all , two sectors , and that is counterfeit goods and cybercrime . And its been terribly successful . I would like to introduce you to Mr. Pringle . Or perhaps I should say , more accurately , S
 e�or Pringle . I was introduced to this bit of kit by a Brazilian cybercriminal . We sat in a car on the Avenue Paulista in S�o Paulo , together . Hooked it up to my laptop , and within about five minutes he had penetrated the computer security system of a major Brazilian bank . It 's really not that difficult . And it 's actually much easier because the fascinating thing about cybercrime is that it 's not so much the technology . The key to cybercrime is what we call social engineering . Or to use the technical term for it , there 's one born every minute . You would not believe how easy it is to persuade people to do things with their computers which are objectively not in their interest . And it was very soon when the cybercriminals learned that the quickest way to do this , of course , the quickest way to a person 's wallet is through the promise sex and love . I expect some of you remember the I-Love-You virus , one of the very great worldwide viruses that came . I was very f
 ortunate when the ILOVEYOU virus came out . Because the first person I received it from was an ex-girlfriend of mine . Now she harbored all sorts of sentiments and emotions towards me at the time , but love was not amongst them . ( Laughter ) And so as soon as I saw this drop into my inbox I dispatched it hastily to the recycle bin , and spared myself a very nasty infection . So , cybercrime , do watch out for it . One thing that we do know that the internet is doing is , the internet is assisting these guys . These are mosquitos who carry the malarial parasite which infests our blood when the mosy has had a free meal at our expense . Now , Artesunate is a very effective drug at destroying the parasite in the early days of infection . But over the past year or so researchers in Cambodia have discovered that what 's happening is the malarial parasite is developing a resistance . And they fear that the reason it 's developing a resistance is because Cambodians ca n't afford the drugs 
 on the commercial market , and so they buy it from the Internet . And these pills contain only low doses of the active ingredient . Which is why things are , the parasite is beginning to develop a resistance . The reason I say this is because we have to know that organized crime impacts all sorts of areas of our lives . You do n't have to sleep with prostitutes or take drugs in order to have a relationship with organized crime . They affect our bank accounts . They affect our communications , our pension funds . They even affect the food that we eat and our governments . This is no longer an issue of Sicilians from Palermo and New York . There is no romance involved with gangsters in the 21st Century . This is a mighty industry and it creates instability and violence wherever it goes . It is a major economic force and we need to take it very , very seriously . It 's been a privilege talking to you . Thank you very much . ( Applause ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/168ted_oliver_sacks_what_hallucination_reveals_about_our_minds.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/168ted_oliver_sacks_what_hallucination_reveals_about_our_minds.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/168ted_oliver_sacks_what_hallucination_reveals_about_our_minds.txt
new file mode 100644
index 0000000..768ce81
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/168ted_oliver_sacks_what_hallucination_reveals_about_our_minds.txt
@@ -0,0 +1,2 @@
+
+We see with the eyes . But we see with the brain as well . And seeing with the brain is often called imagination . And we are familiar with the landscapes of our own imagination , our inscapes . We 've lived with them all our lives . But there are also hallucinations as well . And hallucinations are completely different . They do n't seem to be of our creation . They do n't seem to be under our control . They seem to come from the outside , and [ seem ] to mimic perception . So I am going to be talking about hallucinations . And a particular sort of visual hallucination which I see among my patients . A few months ago I got a phone call from a nursing home where I work . They told me that one of the residents , an old lady in her 90s , was seeing things . And they wondered if she 'd gone bonkers . Or , because she was an old lady , whether she 'd had a stroke , or whether she had Alzheimer 's . And so they asked me if I would come and see Rosalie , the old lady . I went in to see he
 r . It was evident straight away that she was perfectly sane and lucid and of good intelligence . But she 'd been very startled , and very bewildered because she 'd been seeing things . And she told me -- the nurses had n't mentioned this -- that she was blind , that she had been completely blind , from macular degeneration , for five years . But now , for the last few days , she 'd been seeing things . So I said , " What sort of things ? " And she said , " People in Eastern dress , in drapes , walking up and down stairs . A man who turns towards me and smiles . But he has huge teeth on one side of his mouth . Animals too . I see a white building . It 's snowing , a soft snow . I see this horse , with a harness , dragging the snow away . Then , one night , the scene changes . I see cats and dogs walking towards me . They come to a certain point and then stop . Then it changes again . I see a lot of children . They are walking up and down stairs . They wear bright colors , rose and b
 lue , like Eastern dress . " Sometimes , she said , before the people come on she may hallucinate pink and blue squares on the floor , which seem to go up to the ceiling . I said , " Is this like a dream ? " And she said , " No , it 's not like a dream . It 's like a movie . " She said , " It 's got color . It 's got motion . But it 's completely silent , like a silent movie . " And she said that it 's a rather boring movie . She said , " All these people with Eastern dress , walking up and down , very repetitive , very limited . " ( Laughter ) And she has a sense of humor . She knew it was a hallucination . But she was frightened . She 'd lived 95 years and she 'd never had a hallucination before . She said that the hallucinations were unrelated to anything she was thinking or feeling or doing . That they seemed to come on by themselves , or disappear [ by themselves ] . She had no control over them . She said she did n't recognize any of the people or places in the hallucinations 
 . And none of the people or the animals , well , they all seemed oblivious of her . And she did n't know what was going on . She wondered if she was going mad , or losing her mind . Well , I examined her carefully . She was a bright old lady . Perfectly sane . She had no medical problems . She was n't on any medications which could produce hallucinations . But she was blind . And I then said to her , " I think I know what you have . " I said , " There is a special form of visual hallucination which may go with deteriorating vision , or blindness . " " This was originally described , " I said , " back in the 18th century , by a man called Charles Bonnet . And you have Charles Bonnet syndrome . There is nothing wrong with your brain . There is nothing wrong with your mind . You have Charles Bonnet syndrome . " And she was very relieved at this , that there was nothing seriously the matter , and also rather curious . She said , " Who is this Charles Bonnet ? " She said , " Did he have 
 them himself ? " And she said , " Tell all the nurses that I have Charles Bonnet syndrome . " ( Laughter ) " I 'm not crazy . I 'm not demented . I have Charles Bonnet syndrome . " Well , so I did tell the nurses . Now this , for me , is a common situation . I work in old-age homes , largely . I see a lot of elderly people who are hearing impaired or visually impaired . About 10 percent of the hearing impaired people get musical hallucinations . And about 10 percent of the visually impaired people get visual hallucinations . You do n't have to be completely blind , only sufficiently impaired . Now with the original description in the 18th century , Charles Bonnet did not have them . His grandfather had these hallucinations . His grandfather was a magistrate , an elderly man . He 'd had cataract surgery . His vision was pretty poor . And in 1759 he described to his grandson various things he was seeing . The first thing he said was he saw a handkerchief in midair . It was large blue 
 handkerchief with four orange circles . And he knew it was a hallucination . You do n't have handkerchiefs in midair . And then he saw a big wheel in midair . But sometimes he was n't sure whether he was hallucinating or not . Because the hallucinations would fit in the context of the visions . So on one occasion , when his granddaughters were visiting them , he said , " And who are these handsome young men with you ? " And they said , " Alas , Grandpapa , there are no handsome young men . " And then the handsome young men disappeared . It 's typical of these hallucinations that they may come in a flash and disappear in a flash . They do n't usually fade in and out . They are rather sudden . And they change suddenly . Charles Lullin , the grandfather , saw hundreds of different figures , different landscapes of all sorts . On one occasion he saw a man in a bathrobe smoking a pipe , and realized it was himself . That was the only figure he recognized . On one occasion when he was wal
 king in the streets of Paris , he saw -- this was real -- a scaffolding . But when he got back home he saw a miniature of the scaffolding six inches high , on his study table . This repetition of perception is sometimes called palinopsia . With him , and with Rosalie , what seems to be going on -- and Rosalie said , " What 's going on ? " -- and I said that as you lose vision , as the visual parts of the brain are no longer getting any input , they become hyperactive and excitable . And they start to fire spontaneously . And you start to see things . The things you see can be very complicated indeed . With another patient of mine , who , [ like Charles Lullin , still ] had some vision , the vision she had could be disturbing . On one occasion she said she saw a man in a striped shirt in a restaurant . And he turned around . And then he divided into six identical figures in striped shirts , who started walking towards her . And then the six figures came together again , like a concer
 tina . Once , when she was driving , or rather , her husband was driving , the road divided into four . And she felt herself going simultaneously up four roads . She had very mobile hallucinations as well . A lot of them had to do with a car . Sometimes she would see a teenage boy sitting on the hood of the car . He was very tenacious and he moved rather gracefully when the car turned . And then when they came to a stop , the boy would do a sudden vertical take off , 100 foot in the air , and then disappear . Another patient of mine had a different sort of hallucination . This was a woman who did n't have trouble with her eyes , but the visual parts of her brain . A little tumor in the occipital cortex . And , above all , she would see cartoons . These cartoons would be transparent and would cover half the visual field , like a screen . And especially she saw cartoons of Kermit the Frog . ( Laughter ) Now , I do n't watch Sesame Street . But she made a point of saying , " Why Kermit
  ? " She said , " Kermit the Frog means nothing to me . You know , I was wondering about Freudian determinants . Why Kermit ? Kermit the Frog means nothing to me . " She did n't mind the cartoons too much . But what did disturb her was she got very persistent images or hallucinations of faces and as with Rosalie , the faces were often deformed , with very large teeth , or very large eyes . And these frightened her . Well , what is going on with these people ? As a physician , I have to try and define what 's going on , and to reassure people . Especially to reassure them that they 're not going insane . Something like 10 percent , as I said , of visually impaired people get these . But no more than one percent of the people acknowledge them . Because they are afraid they will be seen as insane , or something . And if they do mention them to their own doctors they may be misdiagnosed . In particular , the notion is that if you see things or hear things , you 're going mad . But the p
 sychotic hallucinations are quite different . Psychotic hallucinations , whether they are visual or vocal , they address you . They accuse you . They seduce you . They humiliate you . They jeer at you . You interact with them . There is none of this quality of being addressed with these Charles Bonnet hallucinations . There is a film . You 're seeing a film which has nothing to do with you . Or that 's how people think about it . There is also a rare thing called temporal lobe epilepsy . And sometimes , if one has this , one may feel oneself transported back to a time and place in the past . You 're at a particular road junction . You smell chestnuts roasting . You hear the traffic . All the senses are involved . And you 're waiting for your girl . And it 's that Tuesday evening back in 1982. And the temporal lobe hallucinations are all multisensory hallucinations , full of feeling , full of familiarity , located in space and time , coherent , dramatic . The Charles Bonnet ones are 
 quite different . So in the Charles Bonnet hallucinations , you have all sorts of levels , from the geometrical hallucinations , the pink and blue squares the woman had , up to quite elaborate hallucinations with figures and especially faces . Faces , and sometimes deformed faces , are the single commonest thing in these hallucinations . And one of the second commonest is cartoons . So , what is going on ? Fascinatingly , in the last few years , it 's been possible to do functional brain imagery , to do fMRI on people as they are hallucinating . And in fact , to find that different parts of the visual brain are activated as they are hallucinating . When people have these simple geometrical hallucinations , the primary visual cortex is activated . This is the part of the brain which perceives edges and patterns . You do n't form images with your primary visual cortex . When images are formed , a higher part of the visual cortex is involved in the temporal lobe . And in particular , o
 ne area of the temporal lobe is called the fusiform gyrus . And it 's known that if people have damage in the fusiform gyrus , they maybe lose the ability to recognize faces . But if there is an abnormal activity in the fusiform gyrus , they may hallucinate faces . And this is exactly what you find in some of these people . There is an area in the anterior part of this gyrus where teeth and eyes are represented . And that part of the gyrus is activated when people get the deformed hallucinations . There is another part of the brain which is especially activated when one sees cartoons . It 's activated when one recognizes cartoons , when one draws cartoons , and when one hallucinates them . It 's very interesting that that should be specific . There are other parts of the brain which are specifically involved with the recognition and hallucination of buildings and landscapes . Around 1970 it was found that there were not only particular parts of the brain [ involved ] , but particula
 r cells . " Face cells " were discovered around 1970. And now we know that there are hundreds of other sorts of cells , which can be very very specific . So you may not only have " car " cells , you may have " Aston Martin " cells . ( Laughter ) I saw an Aston Martin this morning . I had to bring it in . And now it 's in there somewhere . ( Laughter ) Now , at this level , in what 's called the inferotemporal cortex , there are only visual images , or figments or fragments . It 's only at higher levels that the other senses join in and there are connections with memory and emotion . And in the Charles Bonnet syndrome you do n't go to those higher levels . You 're in these levels of inferior visual cortex where you have thousands and tens of thousands and millions of images , or figments , or fragmentary figments , all neurally encoded , in particular cells or small clusters of cells . Normally these are all part of the integrated stream of perception , or imagination . And one is no
 t conscious of them . It is only if one is visually impaired , or blind , that the process is interrupted . And instead of getting normal perception , you 're getting an anarchic , convulsive stimulation , or release , of all of these visual cells , in the inferotemporal cortex . So , suddenly you see a face . Suddenly you see a car . Suddenly this , and suddenly that . The mind does its best to organize , and to give some sort of coherence to this . But not terribly successfully . When these were first described it was thought that they could be interpreted like dreams . But in fact people say , " I do n't recognize the people . I ca n't form any associations . " " Kermit means nothing to me . " You do n't get anywhere thinking of them as dreams . Well , I 've more or less said what I wanted . I think I just want to recapitulate and say this is common . Think of the number of blind people . There must be hundreds of thousands of blind people who have these hallucinations , but are 
 too scared to mention them . So this sort of thing needs to be brought into notice , for patients , for doctors , for the public . Finally , I think they are infinitely interesting , and valuable , for giving one some insight as to how the brain works . Charles Bonnet said , 250 years ago -- he wondered how , thinking these hallucinations , how , as he put it , the theater of the mind could be generated by the machinery of the brain . Now , 250 years later , I think we 're beginning to glimpse how this is done . Thanks very much . ( Applause ) Chris Anderson : That was superb . Thank you so much . You speak about these things with so much insight and empathy for your patients . Have you yourself experienced any of the syndromes you write about ? Oliver Sacks : I was afraid you 'd ask that . ( Laughter ) Well , yeah , a lot of them . And actually I 'm a little visually impaired myself . I 'm blind in one eye , and not terribly good in the other . And I see the geometrical hallucinati
 ons . But they stop there . C. A. : And they do n't disturb you ? Because you understand what 's doing it . It does n't make you worried ? O. S. : Well they do n't disturb me any more than my tinnitus . Which I ignore . They occasionally interest me . And I have many pictures of them in my notebooks . I 've gone and had an FMRI myself to see how my visual cortex is taking over . And when I see all these hexagons and complex things , which I also have , in visual migraine , I wonder whether everyone sees things like this , and whether things like cave art , or ornamental art may have been derived from them a bit . C. A. : That was an utterly utterly fascinating talk . Thank you so much for sharing . O. S. : Thank you . Thank you . ( Applause ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/169ted_rebecca_saxe_how_brains_make_moral_judgments.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/169ted_rebecca_saxe_how_brains_make_moral_judgments.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/169ted_rebecca_saxe_how_brains_make_moral_judgments.txt
new file mode 100644
index 0000000..fbd1b9f
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/169ted_rebecca_saxe_how_brains_make_moral_judgments.txt
@@ -0,0 +1,2 @@
+
+Today I 'm going to talk to you about the problem of other minds . And the problem I 'm going to talk about is not the familiar one from philosophy , which is , " How can we know whether other people have minds ? " That is , maybe you have a mind , and everyone else is just a really convincing robot . So that 's a problem in philosophy . But for today 's purposes I 'm going to assume that many people in this audience have a mind , and that I do n't have to worry about this . There is a second problem that is maybe even more familiar to us as parents and teachers and spouses , and novelists . Which is , " Why is it so hard to know what somebody else wants or believes ? " Or perhaps , more relevantly , " Why is it so hard to change what somebody else wants or believes ? " I think novelists put this best . Like Philip Roth , who said , " And yet , what are we to do about this terribly significant business of other people ? So ill equipped are we all , to envision one another 's interio
 r workings and invisible aims . " So as a teacher , and as a spouse , this is , of course , a problem I confront every day . But as a scientist , I 'm interested in a different problem of other minds , and that is the one I 'm going to introduce to you today . And that problem is , " How is it so easy to know other minds ? " So to start with an illustration , you need almost no information , one snapshot of a stranger , to guess what this woman is thinking , or what this man is . And put another way , the crux of the problem is the machine that we use for thinking about other minds , our brain , is made up of pieces , brain cells , that we share with all other animals , with monkeys , and mice , and even sea slugs . And yet , you put them together in a particular network , and what you get is the capacity to write Romeo and Juliet . Or to say , as Alan Greenspan did , " I know you think you understand what you thought I said , but I 'm not sure you realize that what you heard is not
  what I meant . " ( Laughter ) So the job of my field of cognitive neuroscience is to stand with these ideas , one in each hand . And to try to understand how you can put together simple units , simple messages over space and time , in a network , and get this amazing human capacity to think about minds . So I 'm going to tell you three things about this today . Obviously the whole project here is huge . And I 'm going to tell you just our first few steps about the discovery of a special brain region for thinking about other people 's thoughts . Some observations on the slow development of this system as we learn how to do this difficult job . And then finally , to show that some of the differences between people , in how we judge others , can be explained by differences in this brain system . So first , the first thing I want to tell you is that there is a brain region in the human brain , in your brains , whose job it is to think about other people 's thoughts . This is a picture 
 of it . It 's called the Right Temporo-Parietal Junction . It 's above and behind your right ear . And this is the brain region you used when you saw the pictures I showed you , or when you read Romeo and Juliet , or when you tried to understand Alan Greenspan . And you do n't use it for solving any other kinds of logical problems . So this brain region is called the RTPJ . And this picture shows the average activation in a group of what we call typical human adults . They 're MIT undergraduates . ( Laughter ) The second thing I want to say about this brain system is that although we human adults are really good at understanding other minds , we were n't always that way . It takes children a long time to break into the system . I 'm going to show you a little bit of that long , extended process . The first thing I 'm going to show you is a change between age three and five , as kids learn to understand that somebody else can have beliefs that are different from their own . So I 'm g
 oing to show you a five-year-old who is getting a standard kind of puzzle that we call the false belief task . Video : This is the first pirate . His name is Ivan . And you know what pirates really like ? Pirates really like cheese sandwiches . Child : Cheese ? I love cheese ! R. S. : Yeah . So Ivan has this cheese sandwich . and he says " Yum yum yum yum yum ! I really love cheese sandwiches . " And Ivan puts his sandwich over here , on top of the pirate chest . And Ivan says , " You know what ? I need a drink with my lunch . " And so Ivan goes to get a drink . And while Ivan is away the wind comes , and it blows the sandwich down onto the grass . And now , here comes the other pirate . This pirate is called Joshua . And Joshua also really loves cheese sandwiches . So Joshua has a cheese sandwich and he says , " Yum yum yum yum yum ! I love cheese sandwiches . " And he puts his cheese sandwich over here on top of the pirate chest . Child : So , that one is his . R. S. : That one is
  Joshua 's . That 's right . Child : And then his went on the ground . R. S. : That 's exactly right . Child : So he wo n't know which one is his . R. S. : Oh . So now Joshua goes off to get a drink . Ivan comes back and he says , " I want my cheese sandwich . " So which one do you think Ivan is going to take ? Child : I think he is going to take that one . R. S. : Yeah , you think he 's going to take that one ? Alright . Let 's see . Oh yeah , you were right . He took that one . So that 's a five-year-old who clearly understands that other people can have false beliefs and what the consequences are for their actions . Now I 'm going to show you a three-year-old who got the same puzzle . Video : R. S. : And Ivan says , " I want my cheese sandwich . " Which sandwich is he going to take ? Do you think he 's going to take that one ? Let 's see what happens . Let 's see what he does . Here comes Ivan . And he says , " I want my cheese sandwich . " And he takes this one . Uh-oh . Why did
  he take that one ? Child : His was on the grass . R. S. So the three-year-old does two things differently . First he predicts Ivan will take the sandwich that 's really his . And second , when he sees Ivan taking the sandwich where he left his , where we would say he 's taking that one because he thinks it 's his , the three-year-old comes up with another explanation . He 's not taking his own sandwich because he does n't want it , because now it 's dirty , on the ground . So that 's why he 's taking the other sandwich . Now of course , development does n't end at five . And we can see the continuation of this process of learning to think about other people 's thoughts by upping the ante and asking children now , not for an action prediction , but for a moral judgement . So first I 'm going to show you the three-year-old again . Video : R. S. : So is Ivan being mean and naughty for taking Joshua 's sandwich ? Child : Yeah . R. S. : Should Ivan get in trouble for taking Joshua 's sa
 ndwich ? Child : Yeah . R. S. : So it 's maybe not surprising he thinks it was mean of Ivan to take Joshua 's sandwich . Since he thinks Ivan only took Joshua 's sandwich to avoid having to eat his own dirty sandwich . But now I 'm going to show you the five-year-old . Remember the five-year-old completely understood why Ivan took Joshua 's sandwich . Video : R. S. : Was Ivan being mean and naughty for taking Joshua 's sandwich ? Child : Um , yeah . R. S. : And so , it is not until age seven that we get what looks more like an adult response . Video : R. S. : Should Ivan get in trouble for taking Joshua 's sandwich ? Child : No , because the wind should get in trouble . R. S. He says the wind should get in trouble for switching the sandwiches . ( Laughter ) And now what we 've started to do in my lab is to put children into the brain scanner and ask what 's going on in their brain as they develop this ability to think about other people 's thoughts . So the first thing is that in ch
 ildren we see this same brain region , the RTPJ , being used while children are thinking about other people . But it 's not quite like the adult brain . So where as in the adults , as I told you , this brain region is almost completely specialized . It does almost nothing else , except for thinking about other people 's thoughts . In children it 's much less so , when they are age five to eight , the age range of the children I just showed you . And actually if we even look at eight to 11-year-olds , getting into early adolescence , they still do n't have quite an adult-like brain region . And so , what we can see is that over the course of childhood and even into adolescence , both the cognitive system , our mind 's ability to think about other minds , and the brain system that supports it , are continuing , slowly , to develop . But of course , as you 're probably aware , even in adulthood , people differ from one another in how good they are at thinking of other minds , how often
  they do it , and how accurately . And so what we wanted to know was , could differences among adults , in how they think about other people 's thoughts be explained in terms of differences in this brain region . So the first thing that we did is we gave adults a version of the pirate problem that we gave to the kids . And I 'm going to give that to you now . So Grace and her friend are on a tour of a chemical factory and they take a break for coffee . And Grace 's friend asks for some sugar in her coffee . Grace goes to make the coffee and finds by the coffee a pot containing a white powder , which is sugar . But the powder is labeled " Deadly Poison " . So Grace thinks that the powder is a deadly poison . And she puts it in her friend 's coffee . And her friend drinks the coffee , and is fine . How many people think it was morally permissible for Grace to put the powder in the coffee ? Okay . Good . ( Laughter ) So we ask people how much should Grace be blamed in this case , which
  we call a failed attempt to harm . And we can compare that to another case where everything in the real world is the same . The powder is still sugar , but what 's different is what Grace thinks . Now she thinks the powder is sugar . And perhaps unsurprisingly , if Grace thinks the powder is sugar and puts it in her friend 's coffee , people say she deserves no blame at all . Whereas if she thinks the powder was poison , even though it 's really sugar , now people say she deserves a lot of blame , even though what happened in the real world was exactly the same . And in fact they say she deserves more blame in this case , the failed attempt to harm , than in another case , which we call an accident . Where Grace thought the powder was sugar , because it was labeled " sugar " and by the coffee machine , but actually the powder was poison . So even though when the powder was poison , the friend drank the coffee and died , people say Grace deserves less blame in that case , when she i
 nnocently thought it was sugar , than in the other case , where she thought it was poison , and no harm occurred . People , though , disagree a little bit about exactly how much blame Grace should get in the accident case . Some people think she should deserve more blame , and other people less . And what I 'm going to show you is what happened when we look inside the brains of people while they 're making that judgment . So what I 'm showing you , from left to right , is how much activity there was in this brain region . and from top to bottom , how much blame people said that Grace deserved . And what you can see is , on the left when there as very little activity in this brain region , people paid little attention to her innocent belief and said she deserved a lot of blame for the accident . Whereas , on the right , where there was a lot of activity , people payed a lot more attention to her innocent belief , and said she deserved a lot less blame for causing the accident . So th
 at 's good , but of course what we 'd rather is have a way to interfere with function in this brain region , and see if we could change people 's moral judgment . And we do have such a tool . It 's called Trans-Cranial Magnetic Stimulation , or TMS . This is a tool that lets us pass a magnetic pulse through somebody 's skull , into a small region of their brain , and temporarily disorganize the function of the neurons in that region . So I 'm going to show you a demo of this . First I 'm going to show you , to show you that this is a magnetic pulse , I 'm going to show you what happens when you put a quarter on the machine . When you hear clicks we 're turning the machine on . So now I 'm going to apply that same pulse to my brain , to the part of my brain that controls my hand . So there is not physical force , just a magnetic pulse . Video : Woman : Ready ? Rebecca Saxe : Yes . Okay , so it causes a small involuntary contraction in my hand by putting a magnetic pulse in my brain .
  And we can use that same pulse , now applied to the RTPJ , to ask if we can change people 's moral judgments . So these are the judgments I showed you before , people 's normal moral judgments . And then we can apply TMS to the RTPJ and ask how people 's judgments change . And the first thing is , people can still do this task overall . So their judgments of the case when everything was fine remain the same . They say she deserves no blame . But in the case of a failed attempt to harm , where Grace thought that it was poison , although it was really sugar , people now say it was more okay , she deserves less blame for putting the powder in the coffee . And in the case of the accident , where she thought that it was sugar , but it was really poison and so she caused a death , people say that it was less okay , she deserves more blame . So what I 've told you today is that people come , actually , especially well equipped to think about other people 's thoughts . We have a special br
 ain system that lets us think about what other people are thinking . This system takes a long time to develop , slowly throughout the course of childhood , and into early adolescence . And even in adulthood , differences in this brain region can explain differences among adults in how we think about and judge other people . But I want to give the last word back to the novelists . And to Philip Roth , who ended by saying , " The fact remains that getting people right is not what living is all about anyway . It 's getting them wrong that is living . Getting them wrong and wrong and wrong , and then on careful reconsideration , getting them wrong again . " Thank you . ( Applause ) Chris Anderson : When you start talking about using magnetic pulses to change people 's moral judgments , that sounds alarming . ( Laughter ) Please tell me that you 're not taking phone calls from the Pentagon , say . Rebecca Saxe : I 'm not . I mean , they 're calling , but I 'm not taking the call . ( Laug
 hter ) C. A. : They really are calling ? So , then seriously , then seriously , you must lie awake at night sometimes wondering where this work leads . I mean you 're clearly an incredible human being . But someone could take this knowledge and in some future not torture chamber , do acts that people here might be worried about . R. S. : Yeah , we worry about this . So , there is a couple of things to say about TMS . One is that you ca n't be TMSed with out knowing it . So it 's not a surreptitious technology . It 's quite hard actually to get those very small changes . The changes I showed you are impressive to me because of what they tell us about the function of the brain . But they 're small on the scale of the moral judgments that we actually make . And what we changed was not people 's moral judgments when they 're deciding what to do , when they 're making action choices . We change their ability to judge other people 's actions . And so I think of what I 'm doing not so much
  as studying the defendant in a criminal trial , but studying the jury . C. A. : Is your work going to lead to any recommendations in education , to perhaps bring up a generation of kids able to make fairer moral judgments ? R. S. : That 's one of the idealistic hopes . The whole research program here , of studying the distinctive parts of the human brain , is brand new . Until recently what we knew about the brain were the things that any other animal 's brain could do too . So we could study it in animal models . We knew how brains see , and how they control the body , and how they hear and sense . And the whole project of understanding how brains do the uniquely human things , learn language , and abstract concepts , and thinking about other people 's thoughts , that 's brand new . And we do n't know yet , what the implications will be of understanding it . C. A. : So I 've got one last question . There is this thing called the hard problem of consciousness , that puzzles a lot o
 f people . The notion that you can understand why a brain works , perhaps . But why does anyone have to feel anything ? Why does it seem to require these beings who sense things for us to operate ? You 're a brilliant young neuroscientist . I mean , what chances do you think there are that at some time in your career someone , you or someone else , is going to come up with some paradigm shift in understanding what seems an impossible problem . R. S. : I hope they do . And I think they probably wo n't . C. A. : Why ? R. S. : It 's not called the hard problem of consciousness for nothing . ( Laughter ) C. A. : That 's a great answer . Rebecca Saxe , thank you very much . That was fantastic . ( Applause ) 
\ No newline at end of file


[12/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/145ted_benjamin_wallace_on_the_price_of_happiness.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/145ted_benjamin_wallace_on_the_price_of_happiness.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/145ted_benjamin_wallace_on_the_price_of_happiness.txt
new file mode 100644
index 0000000..8c03610
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/145ted_benjamin_wallace_on_the_price_of_happiness.txt
@@ -0,0 +1,2 @@
+
+I 'm just going to play a brief video clip . Video : 50,000 pounds . On the fifth of December 1985 , a bottle of 1787 Lafitte was sold for 105,000 pounds -- nine times the previous world record . Mr. Forbes . The buyer was Kip Forbes , son of one of the most flamboyant millionaires of the 20th century . The original owner of the bottle turned out to be one of the most enthusiastic wine buffs of the 18th century . Ch�teau Lafitte is one of the greatest wines in the world , the prince of any wine cellar . Benjamin Wallace : Now , that 's about all the videotape that remains of an event that set off the longest-running mystery in the modern wine world . And the mystery existed because of a gentleman named Hardy Rodenstock . In 1985 , he announced to his friends in the wine world that he had made this incredible discovery . Some workmen in Paris had broken through a brick wall , and happened upon this hidden cache of wines -- apparently the property of Thomas Jefferson . 1787 , 1784. H
 e would n't reveal the exact number of bottles , he would not reveal exactly where the building was and he would not reveal exactly who owned the building . The mystery persisted for about 20 years . It finally began to get resolved in 2005 because of this guy . Bill Koch is a Florida billionaire who owns four of the Jefferson bottles , and he became suspicious . And he ended up spending over a million dollars and hiring ex-FBI and ex-Scotland Yard agents to try to get to the bottom of this . There 's now ample evidence that Hardy Rodenstock is a con man , and that the Jefferson bottles were fakes . But for those 20 years , an unbelievable number of really eminent and accomplished figures in the wine world were sort of drawn into the orbit of these bottles . I think they wanted to believe that the most expensive bottle of wine in the world must be the best bottle of wine in the world , must be the rarest bottle of wine in the world . I became increasingly , kind of voyeuristically i
 nterested in the question of you know , why do people spend these crazy amounts of money , not only on wine but on lots of things , and are they living a better life than me ? So , I decided to embark on a quest . With the generous backing of a magazine I write for sometimes , I decided to sample the very best , or most expensive , or most coveted item in about a dozen categories , which was a very grueling quest , as you can imagine . ( Laughter ) This was the first one . A lot of the Kobe beef that you see in the U. S. is not the real thing . It may come from Wagyu cattle , but it 's not from the original , Appalachian Hyogo Prefecture in Japan . There are very few places in the U. S. where you can try real Kobe , and one of them is Wolfgang Puck 's restaurant , CUT , in Los Angeles . I went there , and I ordered the eight-ounce rib eye for 160 dollars . And it arrived , and it was tiny . And I was outraged . It was like , 160 dollars for this ? And then I took a bite , and I wish
 ed that it was tinier , because Kobe beef is so rich . It 's like foie gras -- it 's not even like steak . I almost could n't finish it . I was really happy when I was done . ( Laughter ) Now , the photographer who took the pictures for this project for some reason posed his dog in a lot of them , so that 's why you 're going to see this recurring character . Which , I guess , you know , communicates to you that I did not think that one was really worth the price . White truffles . One of the most expensive luxury foods by weight in the world . To try this , I went to a Mario Batali restaurant in Manhattan -- Del Posto . The waiter , you know , came out with the white truffle knob and his shaver , and he shaved it onto my pasta and he said , you know , " Would Signore like the truffles ? " And the charm of white truffles is in their aroma . It 's not in their taste , really . It 's not in their texture . It 's in the smell . These white pearlescent flakes hit the noodles , this haun
 ting , wonderful , nutty , mushroomy smell wafted up . 10 seconds passed and it was gone . And then I was left with these little ugly flakes on my pasta that , you know , their purpose had been served , and so I 'm afraid to say that this was also a disappointment to me . There were several -- several of these items were disappointments . ( Laughter ) Yeah . The magazine would n't pay for me to go there . ( Laughter ) They did give me a tour , though . And this hotel suite is 4,300 square feet . It has 360-degree views . It has four balconies . It was designed by the architect I. M. Pei . It comes with its own Rolls Royce and driver . It comes with its own wine cellar that you can draw freely from . When I took the tour , it actually included some Opus One , I was glad to see . 30,000 dollars for a night in a hotel . This is soap that 's made from silver nanoparticles , which have antibacterial properties . I washed my face with this this morning in preparation for this . And it , y
 ou know , tickled a little bit and it smelled good , but I have to say that nobody here has complimented me on the cleanliness of my face today . ( Laughter ) But then again , nobody has complimented me on the jeans I 'm wearing . These ones GQ did spring for -- I own these -- but I will tell you , not only did I not get a compliment from any of you , I have not gotten a compliment from anybody in the months that I have owned and worn these . I do n't think that whether or not you 're getting a compliment should be the test of something 's value , but I think in the case of a fashion item , an article of clothing , that 's a reasonable benchmark . That said , a lot of work goes into these . They are made from handpicked organic Zimbabwean cotton that has been shuttle loomed and then hand-dipped in natural indigo 24 times . But no compliments . ( Laughter ) Thank you . Armando Manni is a former filmmaker who makes this olive oil from an olive that grows on a single slope in Tuscany .
  And he goes to great lengths to protect the olive oil from oxygen and light . He uses tiny bottles , the glass is tinted , he tops the olive oil off with an inert gas . And he actually -- once he releases a batch of it , he regularly conducts molecular analyses and posts the results online , so you can go online and look at your batch number and see how the phenolics are developing , and , you know , gauge its freshness . I did a blind taste test of this with 20 people and five other olive oils . It tasted fine . It tasted interesting . It was very green , it was very peppery . But in the blind taste test , it came in last . The olive oil that came in first was actually a bottle of Whole Foods 365 olive oil which had been oxidizing next to my stove for six months . ( Laughter ) A recurring theme is that a lot of these things are from Japan -- you 'll start to notice . I do n't play golf , so I could n't actually road test these , but I did interview a guy who owns them . Even the p
 eople who market these clubs -- I mean , they 'll say these have four axis shafts which minimize loss of club speed and thereby drive the ball farther -- but they 'll say , look , you know , you 're not getting 57,000 dollars worth of performance from these clubs . You 're paying for the bling , that they 're encrusted with gold and platinum . The guy who I interviewed who owns them did say that he 's gotten a lot of pleasure out of them , so ... Oh , yeah , you know this one ? This is a coffee made from a very unusual process . The luwak is an Asian Palm Civet . It 's a cat that lives in trees , and at night it comes down and it prowls the coffee plantations . And apparently it 's a very picky eater and it , you know , homes in on only the ripest coffee cherries . And then an enzyme in its digestive tract leeches into the beans , and people with the unenviable job of collecting these cats ' leavings then go through the forest collecting the , you know , results and processing it in
 to coffee -- although you actually can buy it in the unprocessed form . That 's right . Unrelatedly -- ( Laughter ) Japan is doing crazy things with toilets . ( Laughter ) There is now a toilet that has an MP3 player in it . There 's one with a fragrance dispenser . There 's one that actually analyzes the contents of the bowl and transmits the results via email to your doctor . It 's almost like a home medical center -- and that is the direction that Japanese toilet technology is heading in . This one does not have those bells and whistles , but for pure functionality it 's pretty much the best -- the Neorest 600. And to try this -- I could n't get a loaner , but I did go into the Manhattan showroom of the manufacturer , Toto , and they have a bathroom off of the showroom that you can use , which I used . It 's fully automated -- you walk towards it , and the seat lifts . The seat is preheated . There 's a water jet that cleans you . There 's an air jet that dries you . You get up ,
  it flushes by itself . The lid closes , it self-cleans . Not only is it a technological leap forward , but I really do believe it 's a bit of a cultural leap forward . I mean , a no hands , no toilet paper toilet . And I want to get one of these . ( Laughter ) This was another one I could not get a loaner of . Tom Cruise supposedly owns this bed . There 's a little plaque on the end that , you know , each buyer gets their name engraved on it . ( Laughter ) To try this one , the maker of it let me and my wife spend the night in the Manhattan showroom . Lights glaring in off the street , and we had to hire a security guard and all these things . But anyway , we had a great night 's sleep . And you spend a third of your life in bed . I do n't think it 's that bad of a deal . ( Laughter ) This was a fun one . This is the fastest street-legal car in the world and the most expensive production car . I got to drive this with a chaperone from the company , a professional race car driver , 
 and we drove around the canyons outside of Los Angeles and down on the Pacific Coast Highway . And , you know , when we pulled up to a stoplight the people in the adjacent cars kind of gave us respectful nods . And it was really amazing . It was such a smooth ride . Most of the cars that I drive , if I get up to 80 they start to rattle . I switched lanes on the highway and the driver , this chaperone , said , " You know , you were just going 110 miles an hour . " And I had no idea that I was one of those obnoxious people you occasionally see weaving in and out of traffic , because it was just that smooth . And if I was a billionaire , I would get one . ( Laughter ) This is a completely gratuitous video I 'm just going to show of one of the pitfalls of advanced technology . This is Tom Cruise arriving at the " Mission : Impossible III " premiere . When he tries to open the door , you could call it " Mission : Impossible IV . " There was one object that I could not get my hands on , a
 nd that was the 1947 Cheval Blanc . The '47 Cheval Blanc is probably the most mythologized wine of the 20th century . And Cheval Blanc is kind of an unusual wine for Bordeaux in having a significant percentage of the Cabernet Franc grape . And 1947 was a legendary vintage , especially in the right bank of Bordeaux . And just together , that vintage and that chateau took on this aura that eventually kind of gave it this cultish following . But it 's 60 years old . There 's not much of it left . What there is of it left you do n't know if it 's real -- it 's considered to be the most faked wine in the world . Not that many people are looking to pop open their one remaining bottle for a journalist . So , I 'd about given up trying to get my hands on one of these . I 'd put out feelers to retailers , to auctioneers , and it was coming up empty . And then I got an email from a guy named Bipin Desai . Bipin Desai is a UC Riverside theoretical physicist who also happens to be the preeminen
 t organizer of rare wine tastings , and he said , " I 've got a tasting coming up where we 're going to serve the '47 Cheval Blanc . " And it was going to be a double vertical -- it was going to be 30 vintages of Cheval Blanc , and 30 vintages of Yquem . And it was an invitation you do not refuse . I went . It was three days , four meals . And at lunch on Saturday , we opened the '47 . And you know , it had this fragrant softness , and it smelled a little bit of linseed oil . And then I tasted it , and it , you know , had this kind of unctuous , porty richness , which is characteristic of that wine -- that it sort of resembles port in a lot of ways . There were people at my table who thought it was , you know , fantastic . There were some people who were a little less impressed . And I was n't that impressed . And I do n't -- call my palate a philistine palate -- so it does n't necessarily mean something that I was n't impressed , but I was not the only one there who had that reacti
 on . And it was n't just to that wine . Any one of the wines served at this tasting , if I 'd been served it at a dinner party , it would have been , you know , the wine experience of my lifetime , and incredibly memorable . But drinking 60 great wines over three days , they all just blurred together , and it became almost a grueling experience . And I just wanted to finish by mentioning a very interesting study which came out earlier this year from some researchers at Stanford and Caltech . And they gave subjects the same wine , labeled with different price tags . A lot of people , you know , said that they liked the more expensive wine more -- it was the same wine , but they thought it was a different one that was more expensive . But what was unexpected was that these researchers did MRI brain imaging while the people were drinking the wine , and not only did they say they enjoyed the more expensively labeled wine more -- their brain actually registered as experiencing more pleas
 ure from the same wine when it was labeled with a higher price tag . Thank you . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/146ted_bono_s_call_to_action_for_africa.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/146ted_bono_s_call_to_action_for_africa.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/146ted_bono_s_call_to_action_for_africa.txt
new file mode 100644
index 0000000..c3bd2cd
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/146ted_bono_s_call_to_action_for_africa.txt
@@ -0,0 +1,2 @@
+
+Well , as Alexander Graham Bell famously said on his first successful telephone call , " Hello , is that Domino 's Pizza ? " ( Laughter ) I just really want to thank you very much . As another famous man , Jerry Garcia , said , " What a strange , long trip . " And he should have said , " What a strange , long trip it 's about to become . " At this very moment , you are viewing my upper half . My lower half is appearing at a different conference -- ( Laughter ) in a different country . You can , it turns out , be in two places at once . But still , I 'm sorry I ca n't be with you in person . I 'll explain at another time . And though I 'm a rock star , I just want to assure you that none of my wishes will include a hot tub . But what really turns me on about technology is not just the ability to get more songs on mp3 players . The revolution -- this revolution -- is much bigger than that . I hope , I believe . What turns me on about the digital age , what excited me personally , is t
 hat you have closed the gap between dreaming and doing . You see , it used to be that if you wanted to make a record of a song , you needed a studio and a producer . Now , you need a laptop . If you wanted to make a film , you needed a mass of equipment and a Hollywood budget . Now , you need a camera that fits in your palm , and a couple of bucks for a blank DVD . Imagination has been decoupled from the old constraints . And that really , really excites me . I 'm excited when I glimpse that kind of thinking writ large . What I would like to see is idealism decoupled from all constraints . Political , economic , psychological , whatever . The geopolitical world has got a lot to learn from the digital world . From the ease with which you swept away obstacles that no one knew could even be budged . And that 's actually what I 'd like to talk about today . First , though , I should probably explain why , and how , I got to this place . It 's a journey that started 20 years ago . You ma
 y remember that song , " We Are the World , " or , " Do They Know It 's Christmas ? " Band Aid , Live Aid . Another very tall , grizzled rock star , my friend Sir Bob Geldof , issued a challenge to " feed the world . " It was a great moment , and it utterly changed my life . That summer , my wife , Ali , and myself went to Ethiopia . We went on the quiet to see for ourselves what was going on . We lived in Ethiopia for a month , working at an orphanage . The children had a name for me . They called me , " The girl with the beard . " ( Laughter ) Do n't ask . Anyway , we found Africa to be a magical place . Big skies , big hearts , big , shining continent . Beautiful , royal people . Anybody who ever gave anything to Africa got a lot more back . Ethiopia did n't just blow my mind ; it opened my mind . Anyway , on our last day at this orphanage a man handed me his baby and said , " Would you take my son with you ? " He knew , in Ireland , that his son would live , and that in Ethiopia
  , his son would die . It was the middle of that awful famine . Well , I turned him down . And it was a funny kind of sick feeling , but I turned him down . And it 's a feeling I ca n't ever quite forget . And in that moment , I started this journey . In that moment , I became the worst thing of all : I became a rock star with a cause . Except this is n't the cause , is it ? Six and a half thousand Africans dying every single day from AIDS -- a preventable , treatable disease -- for lack of drugs we can get in any pharmacy . That 's not a cause . That 's an emergency . 11 million AIDS orphans in Africa , 20 million by the end of the decade . That 's not a cause . That 's an emergency . Today , every day , 9,000 more Africans will catch HIV because of stigmatization and lack of education . That 's not a cause . That 's an emergency . So what we 're talking about here is human rights . The right to live like a human . The right to live , period . And what we 're facing in Africa is an
  unprecedented threat to human dignity and equality . The next thing I 'd like to be clear about is what this problem is , and what this problem is n't . Because this is not all about charity . This is about justice . Really . This is not about charity . This is about justice . That 's right . And that 's too bad , because we 're very good at charity . Americans , like Irish people , are good at it . Even the poorest neighborhoods give more than they can afford . We like to give , and we give a lot . Look at the response to the tsunami , it 's inspiring . But justice is a tougher standard than charity . You see , Africa makes a fool of our idea of justice . It makes a farce of our idea of equality . It mocks our pieties . It doubts our concern . It questions our commitment . Because there is no way we can look at what 's happening in Africa , and if we 're honest , conclude that it would ever be allowed to happen anywhere else . As you heard in the film , anywhere else , not here . 
 Not here , not in America , not in Europe . In fact , a head of state that you 're all familiar with admitted this to me . And it 's really true . There is no chance this kind of hemorrhaging of human life would be accepted anywhere else other than Africa . Africa is a continent in flames . And deep down , if we really accepted that Africans were equal to us , we would all do more to put the fire out . We 're standing around with watering cans , when what we really need is the fire brigade . You see , it 's not as dramatic as the tsunami . It 's crazy , really , when you think about it . Does stuff have to look like an action movie these days to exist in the front of our brain ? The slow extinguishing of countless lives is just not dramatic enough , it would appear . Catastrophes that we can avert are not as interesting as ones we could avert . Funny , that . Anyway , I believe that that kind of thinking offends the intellectual rigor in this room . Six and a half thousand people dy
 ing a day in Africa may be Africa 's crisis , but the fact that it 's not on the nightly news , that we in Europe , or you in America , are not treating it like an emergency -- I want to argue with you tonight that that 's our crisis . I want to argue that though Africa is not the front line in the war against terror , it could be soon . Every week , religious extremists take another African village . They 're attempting to bring order to chaos . Well , why are n't we ? Poverty breeds despair . We know this . Despair breeds violence . We know this . In turbulent times , is n't it cheaper , and smarter , to make friends out of potential enemies than to defend yourself against them later ? The war against terror is bound up in the war against poverty . And I did n't say that . Colin Powell said that . Now when the military are telling us that this is a war that cannot be won by military might alone , maybe we should listen . There 's an opportunity here , and it 's real . It 's not sp
 in . It 's not wishful thinking . The problems facing the developing world afford us in the developed world a chance to re-describe ourselves to the world . We will not only transform other people 's lives , but we will also transform the way those other lives see us . And that might be smart in these nervous , dangerous times . Do n't you think that on a purely commercial level , that anti-retroviral drugs are great advertisements for Western ingenuity and technology ? Does n't compassion look well on us ? And let 's cut the crap for a second . In certain quarters of the world , brand EU , brand USA , is not at its shiniest . The neon sign is fizzing and cracking . Someone 's put a brick through the window . The regional branch managers are getting nervous . Never before have we in the west been so scrutinized . Our values : do we have any ? Our credibility ? These things are under attack around the world . Brand USA could use some polishing . And I say that as a fan , you know ? A
 s a person who buys the products . But think about it . More anti-retrovirals make sense . But that 's just the easy part , or ought to be . But equality for Africa -- that 's a big , expensive idea . You see , the scale of the suffering numbs us into a kind of indifference . What on earth can we all do about this ? Well , much more than we think . We ca n't fix every problem , but the ones we can , I want to argue , we must . And because we can , we must . This is the straight truth , the righteous truth . It is not a theory . The fact is that ours is the first generation that can look disease and extreme poverty in the eye , look across the ocean to Africa , and say this , and mean it . We do not have to stand for this . A whole continent written off -- we do not have to stand for this . ( Applause ) And let me say this without a trace of irony -- before I back it up to a bunch of ex-hippies . Forget the '60s . We can change the world . I ca n't , you ca n't , as individuals , but
  we can change the world . I really believe that , the people in this room . Look at the Gates Foundation . They 've done incredible stuff , unbelievable stuff . But working together , we can actually change the world . We can turn the inevitable outcomes , and transform the quality of life for millions of lives who look and feel rather like us , when you 're up close . I 'm sorry to laugh here , but you do look so different than you did in Haight-Ashbury in the '60s . ( Laughter ) But I want to argue that this is the moment that you are designed for . It is the flowering of the seeds you planted in earlier , headier days. Ideas that you gestated in your youth . This is what excites me . This room was born for this moment , is really what I want to say to you tonight . Most of you started out wanting to change the world , did n't you ? Most of you did , the digital world . Well , now , actually because of you , it is possible to change the physical world . It 's a fact . Economists 
 confirm it , and they know much more than I do . So why , then , are we not pumping our fists into the air ? Probably because when we admit we can do something about it , we 've got to do something about it . It is a pain in the arse . This equality business is actually a pain in the arse . But for the first time in history , we have the technology , we have the know-how , we have the cash , we have the life-saving drugs . Do we have the will ? I hope this is obvious , but I 'm not a hippie . And I 'm not really one for the warm , fuzzy feeling . I do not have flowers in my hair . Actually , I come from punk rock . The Clash wore big army boots , not sandals . But I know toughness when I see it . And for all the talk of peace and love on the West Coast , there was muscle to the movement that started out here . You see , idealism detached from action is just a dream . But idealism allied with pragmatism , with rolling up your sleeves and making the world bend a bit , is very exciting
  . It 's very real . It 's very strong . And it 's very present in a crowd like you . Last year at DATA , this organization I helped set up , we launched a campaign to summon this spirit in the fight against AIDS and extreme poverty . We 're calling it the ONE Campaign . It 's based on our belief that the action of one person can change a lot , but the actions of many coming together as one can change the world . Well , we feel that now is the time to prove we 're right . There are moments in history when civilization redefines itself . We believe this is one . We believe that this could be the time when the world finally decides that the wanton loss of life in Africa is just no longer acceptable . This could be the time that we finally get serious about changing the future for most people who live on planet Earth . Momentum has been building . Lurching a little , but it 's building . This year is a test for us all , especially the leaders of the G8 nations , who really are on the l
 ine here , with all the world in history watching . I have been , of late , disappointed with the Bush Administration . They started out with such promise on Africa . They made some really great promises , and actually have fulfilled a lot of them . But some of them they have n't . They do n't feel the push from the ground , is the truth . But my disappointment has much more perspective when I talk to American people , and I hear their worries about the deficit , and the fiscal well-being of their country . I understand that . But there 's much more push from the ground than you 'd think , if we got organized . What I try to communicate , and you can help me if you agree , is that aid for Africa is just great value for money at a time when America really needs it . Putting it in the crassest possible terms , the investment reaps huge returns . Not only in lives saved , but in goodwill , stability and security that we 'll gain . So this is what I hope that you will do , if I could be
  so bold , and not have it deducted from my number of wishes . ( Laughter ) What I hope is that beyond individual merciful acts , that you will tell the politicians to do right by Africa , by America and by the world . Give them permission , if you like , to spend their political capital and your financial capital , your national purse on saving the lives of millions of people . That 's really what I would like you to do . Because we also need your intellectual capital : your ideas , your skills , your ingenuity . And you , at this conference , are in a unique position . Some of the technologies we 've been talking about , you invented them , or at least revolutionized the way that they 're used . Together you have changed the zeitgeist from analog to digital , and pushed the boundaries . And we 'd like you to give us that energy . Give us that kind of dreaming , that kind of doing . As I say , there 're two things on the line here . There 's the continent Africa . But there 's also
  our sense of ourselves . People are starting to figure this out . Movements are springing up . Artists , politicians , pop stars , priests , CEOs , NGOs , mothers ' unions , student unions . A lot of people are getting together , and working under this umbrella I told you about earlier , the ONE Campaign . I think they just have one idea in their mind , which is , where you live in the world , should not determine whether you live in the world . ( Applause ) History , like God , is watching what we do . When the history books get written I think our age will be remembered for three things . Really , it 's just three things this whole age will be remembered for . The digital revolution , yes . The war against terror , yes . And what we did or did not do to put out the fires in Africa . Some say we ca n't afford to . I say we ca n't afford not to . Thank you , thank you very much . ( Applause ) Okay , my three wishes . The ones that TED has offered to grant . You see , if this is tru
 e , and I believe it is , that the digital world you all created has uncoupled the creative imagination from the physical constraints of matter . This should be a piece of piss . ( Laughter ) I should add that this started out as a much longer list of wishes . Most of them impossible , some of them impractical and one or two of them certainly immoral . ( Laughter ) This business , it gets to be addictive , you know what I mean , when somebody else is picking up the tab . Anyway , here 's number one . I wish for you to help build a social movement of more than one million American activists for Africa . That is my first wish . I believe it 's possible . A few minutes ago , I talked about all the citizens ' campaigns that are springing up . You know , there 's lots out there . And with this one campaign as our umbrella , my organization , DATA , and other groups , have been tapping into the energy and the enthusiasm that 's out there from Hollywood into the heartland of America . We k
 now there 's more than enough energy to power this movement . We just need your help in making it happen . We want all of you here , church America , corporate America , Microsoft America , Apple America , Coke America , Pepsi America , nerd America , noisy America . We ca n't afford to be cool and sit this one out . I do believe if we build a movement that 's one million Americans strong , we 're not going to be denied . We will have the ear of Congress . We 'll be the first page in Condi Rice 's briefing book , and right into the Oval Office . If there 's one million Americans -- and I really know this -- who are ready to make phone calls , who are ready to be on email . I am absolutely sure that we can actually change the course of history , literally , for the continent of Africa . Anyway , so I 'd like your help in getting that signed up . I know John Gage and Sun Microsystems are already on board for this , but there 's lots of you we 'd like to talk to . Right , my second wis
 h , number two . I would like one media hit for every person on the planet who is living on less than one dollar a day . That 's one billion media hits . Could be on Google , could be on AOL . Steve Case , Larry , Sergey -- they 've done a lot already . It could be NBC . It could be ABC . Actually we 're talking to ABC today about the Oscars . We have a film , produced by Jon Kamen at Radical Media . But you know , we want , we need some airtime for our ideas . We need to get the math , we need to get the statistics out to the American people . I really believe that old Truman line , that if you give the American people the facts , they 'll do the right thing . And , the other thing that 's important , is that this is not Sally Struthers . This has to be described as an adventure , not a burden . ( Video ) : One by one they step forward , a nurse , a teacher , a homemaker , and lives are saved . The problem is enormous . Every three seconds one person dies . Another three seconds , 
 one more . The situation is so desperate in parts of Africa , Asia , even America that aid groups , just as they did for the tsunami are uniting as one , acting as one . We can beat extreme poverty , starvation , AIDS . But we need your help . One more person , letter , voice will mean the difference between life and death for millions of people . Please join us by working together . Americans have an unprecedented opportunity . We can make history . We can start to make poverty history . One , by one , by one . Please visit ONE at this address . We 're not asking for your money . We 're asking for your voice . Bono : All right . I wish for TED to truly show the power of information . Its power to rewrite the rules and transform lives , by connecting every hospital , health clinic and school in one African country . And I would like it to be Ethiopia . I believe we can connect every school in Ethiopia , every health clinic , every hospital . We can connect to the Internet . That is 
 my wish , my third wish . I think it 's possible . I think we have the money and brains in the room to do that . And that would be a mind-blowing wish to come true . I 've been to Ethiopia , as I said earlier . It 's actually where it all started for me . The idea that the Internet , which changed all of our lives , can transform a country -- and a continent that has hardly made it to analog , let alone digital -- blows my mind . But it did n't start out that way . The first long-distance line from Boston to New York was used in 1885 on the phone . It was just nine years later that Addis Ababa was connected by phone to Harare , which is 500 kilometers away . Since then , not that much has changed . The average waiting time to get a land line in Ethiopia is actually about seven or eight years . But wireless technology was n't dreamt up then . Anyway , I 'm Irish , and as you can see , I know how important talking is . Communication is very important for Ethiopia -- will transform the
  country . Nurses getting better training , pharmacists being able to order supplies , doctors sharing their expertise in all aspects of medicine . It 's a very , very good idea to get them wired . And that is my third and final wish for you at the TED conference . Thank you very much once again . ( Applause ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/147ted_dan_dennett_cute_sexy_sweet_funny.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/147ted_dan_dennett_cute_sexy_sweet_funny.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/147ted_dan_dennett_cute_sexy_sweet_funny.txt
new file mode 100644
index 0000000..35e124d
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/147ted_dan_dennett_cute_sexy_sweet_funny.txt
@@ -0,0 +1,2 @@
+
+I\u2019m going around the world giving talks about Darwin , and usually what I\u2019m talking about is Darwin\u2019s strange inversion of reasoning . Now that title , that phrase , comes from a critic , an early critic , and this is a passage that I just love , and would like to read for you . " In the theory with which we have to deal , Absolute Ignorance is the artificer ; so that we may enunciate as the fundamental principle of the whole system , that , in order to make a perfect and beautiful machine , it is not requisite to know how to make it . This proposition will be found on careful examination , to express , in condensed form , the essential purport of the Theory , and to express in a few words all Mr. Darwin\u2019s meaning ; who , by a strange inversion of reasoning , seems to think Absolute Ignorance fully qualified to take the place of Absolute Wisdom in the achievements of creative skill . " Exactly . Exactly . And it is a strange inversion . A creationist pamphlet has this wonder
 ful page in it : " Test Two : Do you know of any building that didn\u2019t have a builder ? Yes No. Do you know of any painting that didn\u2019t have a painter ? Yes No. Do you know of any car that didn\u2019t have a maker ? Yes No. If you answered " YES " for any of the above , give details . " A-ha ! I mean , it really is a strange inversion of reasoning . You would have thought it stands to reason that design requires an intelligent designer . But Darwin shows that it\u2019s just false . Today , though , I\u2019m going to talk about Darwin\u2019s other strange inversion , which is equally puzzling at first , but in some ways just as important . It stands to reason that we love chocolate cake because it is sweet . Guys go for girls like this because they are sexy . We adore babies because they\u2019re so cute . And , of course , we are amused by jokes because they are funny . This is all backwards . It is . And Darwin shows us why . Let\u2019s start with sweet . Our sweet tooth is basically an evolved su
 gar detector , because sugar is high energy , and it\u2019s just been wired up to the preferer , to put it very crudely , and that\u2019s why we like sugar . Honey is sweet because we like it , not " we like it because honey is sweet . " There\u2019s nothing intrinsically sweet about honey . If you looked at glucose molecules till you were blind , you wouldn\u2019t see why they tasted sweet . You have to look in our brains to understand why they\u2019re sweet . So if you think first there was sweetness , and then we evolved to like sweetness , you\u2019ve got it backwards ; that\u2019s just wrong . It\u2019s the other way round . Sweetness was born with the wiring which evolved . And there\u2019s nothing intrinsically sexy about these young ladies . And it\u2019s a good thing that there isn\u2019t , because of there were , then Mother Nature would have a problem : How on earth do you get chimps to mate ? Now you might think , ah , there\u2019s a solution : hallucinations . That would be one way of doing it , but there
 \u2019s a quicker way . Just wire the chimps up to love that look , and apparently they do . That\u2019s all there is to it . Over six million years , we and the chimps evolved our different ways . We became bald-bodied , oddly enough ; for one reason or another , they didn\u2019t . If we hadn\u2019t , then probably this would be the height of sexiness . Our sweet tooth is an evolved and instinctual preference for high-energy food . It wasn\u2019t designed for chocolate cake . Chocolate cake is a supernormal stimulus . The term is owed to Niko Tinbergen , who did his famous experiments with gulls , where he found that that orange spot on the gull\u2019s beak -- if he made a bigger , oranger spot the gull chicks would peck at it even harder . It was a hyperstimulus for them , and they loved it . What we see with , say , chocolate cake is it\u2019s a supernormal stimulus to tweak our design wiring . And there are lots of supernormal stimuli ; chocolate cake is one . There 's lots of supernormal stimuli fo
 r sexiness . And there 's even supernormal stimuli for cuteness . Here\u2019s a pretty good example . It\u2019s important that we love babies , and that we not be put off by , say , messy diapers . So babies have to attract our affection and our nurturing , and they do . And , by the way , a recent study shows that mothers prefer the smell of the dirty diapers of their own baby . So nature works on many levels here . But now , if babies didn\u2019t look the way they do , if babies looked like this , that\u2019s what we would find adorable , that\u2019s what we would find -- we would think , oh my goodness , do I ever want to hug that . This is the strange inversion . Well now , finally what about funny . My answer is , it\u2019s the same story , the same story . This is the hard one , the one that isn\u2019t obvious . That\u2019s why I leave it to the end . And I won\u2019t be able to say too much about it . But you have to think evolutionarily , you have to think , what hard job that has to be done -- it\u2019s
  dirty work , somebody\u2019s got to do it -- is so important to give us such a powerful , inbuilt reward for it when we succeed . Now , I think we 've found the answer , I and a few of my colleagues . It\u2019s a neural system that\u2019s wired up to reward the brain for doing a grubby clerical job . Our bumper sticker for this view is that this is the joy of debugging . Now I\u2019m not going to have time to spell it all out , but I\u2019ll just say that only some kinds of debugging get the reward . And what we\u2019re doing is we\u2019re using humor as a sort of neuroscientific probe by switching humor on and off , by turning the knob on a joke -- now it\u2019s not funny ... oh , now it\u2019s funnier ... now we\u2019ll turn a little bit more ... now it\u2019s not funny -- in this way , we can actually learn something about the architecture of the brain , the functional architecture of the brain . Matthew Hurley is the first author of this . We call it the Hurley Model . He\u2019s a computer scientist , Reginald Ad
 ams a psychologist , and there I am , and we\u2019re putting this together into a book . Thank you very much . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/148ted_daniel_kraft_invents_a_better_way_to_harvest_bone_marrow.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/148ted_daniel_kraft_invents_a_better_way_to_harvest_bone_marrow.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/148ted_daniel_kraft_invents_a_better_way_to_harvest_bone_marrow.txt
new file mode 100644
index 0000000..5c8b8aa
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/148ted_daniel_kraft_invents_a_better_way_to_harvest_bone_marrow.txt
@@ -0,0 +1,2 @@
+
+So I am a pediatric cancer doctor and stem-cell researcher at Stanford University where my clinical focus has been bone marrow transplantation . Now , inspired by Jill Bolte Taylor last year , I did n't bring a human brain , but I did bring a liter of bone marrow . And bone marrow is actually what we use to save the lives of tens of thousands of patients , most of whom have advanced malignancies like leukemia and lymphoma and some other diseases . So , a few years ago , I 'm doing my transplant fellowship at Stanford . I 'm in the operating room . We have Bob here , who is a volunteer donor . We 're sending his marrow across the country to save the life of a child with leukemia . So actually how do we harvest this bone marrow ? Well we have a whole O. R. team , general anesthesia , nurses , and another doctor across from me . Bob 's on the table , and we take this sort of small needle , you know , not too big . And the way we do this is we basically place this through the soft tissu
 e , and kind of punch it into the hard bone , into the tuchus -- that 's a technical term -- and aspirate about 10 mls of bone marrow out , each time , with a syringe . And hand it off to the nurse . She squirts it into a tin . Hands it back to me . And we do that again and again . About 200 times usually . And by the end of this my arm is sore , I 've got a callus on my hand . Let alone Bob , Whose rear end looks something more like this , like swiss cheese . So I 'm thinking , you know , this procedure has n't changed in about 40 years . And there is probably a better way to do this . So I thought of a minimally invasive approach . And a new device that we call the Marrow Miner . This is it . And the Marrow Miner , the way it works is shown here . Our standard see-through patient . Instead of entering the bone dozens of times , we enter just once , into the front of the hip or the back of the hip . And we have a flexible , powered catheter with a special wire loop tip that stays i
 nside the crunchy part of the marrow and follows the contours of the hip , as it moves around . So it enables you to very rapidly aspirate , or suck out , rich bone marrow very quickly through one hole . We can do multiple passes through that same entry . No robots required . And , so , very quickly , Bob can just get one puncture , local anesthesia , and do this harvest as an outpatient . So I did a few prototypes . I got a small little grant at Stanford . And played around with this a little bit . And our team members developed this technology . And eventually we got two large animals , and pig studies . And we found , to our surprise , that we not only got bone marrow out , but we got 10 times the stem cell activity in the marrow from the Marrow Miner , compared to the normal device . This device was just FDA approved in the last year . Here is a live patient . You can see it following the flexible curves around . There will be be two passes here , in the same patient , from the 
 same hole . This was done under local anesthesia , as an outpatient . And we got , again , about three to six times more stem cells than the standard approach done on the same patient . So why should you care ? Bone marrow is a very rich source of adult stem cells . You all know about embryonic stem cells . They 've got great potential but have n't yet entered clinical trials . Adult stem cells are throughout our body , including the blood-forming stem cells in our bone marrow . Which we 've been using as a form of stem-cell therapy for over 40 years . In the last decade there 's been an explosion of use of bone marrow stem cells to treat the patient 's other diseases such as heart disease , vascular disease , orthopedics , tissue engineering , even in neurology to treat Parkinson 's , and diabetes . We 've just come out , we 're commercializing , this year , generation 2.0 of the Marrow Miner . The hope is is that this gets more stem cells out . Which translates to better outcomes 
 . It may encourage more people to sign up to be potential live saving bone marrow donors . It may even enable you to bank your own marrow stem cells , when you 're younger and healthier , to use in the future , should you need it . And ultimately -- and here 's a picture of our bone marrow transplant survivors , who come together for a reunion each year at Stanford . Hopefully this technology will let us have more of these survivors in the future . Thanks . ( Applause ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/149ted_dave_eggers_makes_his_ted_prize_wish_once_upon_a_school.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/149ted_dave_eggers_makes_his_ted_prize_wish_once_upon_a_school.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/149ted_dave_eggers_makes_his_ted_prize_wish_once_upon_a_school.txt
new file mode 100644
index 0000000..8f43c3f
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/149ted_dave_eggers_makes_his_ted_prize_wish_once_upon_a_school.txt
@@ -0,0 +1,2 @@
+
+Thank you so much everyone from TED , and Chris and Amy in particular . I cannot believe I 'm here . I have not slept in weeks . Neil and I were sitting there comparing how little we 've slept in anticipation for this . I 've never been so nervous -- and I do this when I 'm nervous , I just realized . So , I 'm going to talk about sort of what we did at this organization called 826 Valencia , and then I 'm going to talk about how we all might join in and do similar things . Back in about 2000 , I was living in Brooklyn , I was trying to finish my first book , I was wandering around dazed every day because I wrote from 12 a. m. to 5 a. m. So I would walk around in a daze during the day . I had no mental acuity to speak of during the day , but I had flexible hours . In the Brooklyn neighborhood that I lived in , Park Slope , there are a lot of writers -- it 's like a very high per-capita ratio of writers to normal people . Meanwhile , I had grown up around a lot of teachers . My mom w
 as a teacher , my sister became a teacher and after college so many of my friends went into teaching . And so I was always hearing them talk about their lives and how inspiring they were , and they were really sort of the most hard-working and constantly inspiring people I knew . But I knew so many of the things they were up against , so many of the struggles they were dealing with . And one of them was that so many of my friends that were teaching in city schools were having trouble with their students keeping up at grade level , in their reading and writing in particular . Now , so many of these students had come from households where English is n't spoken in the home , where many of them have different special needs , learning disabilities . And of course they 're working in schools which sometimes and very often are under-funded . And so they would talk to me about this and say , " You know , what we really need is just more people , more bodies , more one-on-one attention , mor
 e hours , more expertise from people that have skills in English and can work with these students one-on-one . " Now , I would say , " Well , why do n't you just work with them one-on-one ? " And they would say , " Well , we have five classes of 30 to 40 students each . This can lead up to 150 , 180 , 200 students a day . How can we possibly give each student even one hour a week of one-on-one attention ? " You 'd have to greatly multiply the workweek and clone the teachers . And so we started talking about this . And at the same time , I thought about this massive group of people I knew : writers , editors , journalists , graduate students , assistant professors , you name it . All these people that had sort of flexible daily hours and an interest in the English word -- I hope to have an interest in the English language , but I 'm not speaking it well right now . I 'm trying . That clock has got me . But everyone that I knew had an interest in the primacy of the written word in ter
 ms of nurturing a democracy , nurturing an enlightened life . And so they had , you know , their time and their interest , but at the same time there was n't a conduit that I knew of in my community to bring these two communities together . So when I moved back to San Francisco , we rented this building . And the idea was to put McSweeney 's -- McSweeney 's Quarterly , that we published twice or three times a year , and a few other magazines -- we were going to move it into an office for the first time . It used to be in my kitchen in Brooklyn . We were going to move it into an office , and we were going to actually share space with a tutoring center . So we thought , " We 'll have all these writers and editors and everybody -- sort of a writing community coming into the office every day anyway , why do n't we just open up the front of the building for students to come in there after school , get extra help on their written homework , so you have basically no border between these tw
 o communities ? " So the idea was that we would be working on whatever we 're working on , at 2:30 the students flow in and you put down what you 're doing , or you trade , or you work a little bit later or whatever it is . You give those hours in the afternoon to the students in the neighborhood . So , we had this place , we rented it , the landlord was all for it . We did this mural , that 's a Chris Ware mural , that basically explains the entire history of the printed word , in mural form -- it takes a long time to digest and you have to stand in the middle of the road . So we rented this space . And everything was great except the landlord said , " Well , the space is zoned for retail ; you have to come up with something . " You 've gotta sell something . You ca n't just have a tutoring center . " So we thought , " Ha ha ! Really ! " And we could n't think of anything necessarily to sell , but we did all the necessary research . It used to be a weight room , so there were rubbe
 r floors below , acoustic tile ceilings , fluorescent lights . We took all that down , and we found beautiful wooden floors , whitewashed beams and it had the look -- while we were renovating this place , somebody said , " You know , it really kind of looks like the hull of a ship . " And we looked around and somebody else said , " Well , you should sell supplies to the working buccaneer . " And so this is what we did . So it made everybody laugh , and we said , " There 's a point to that . Let 's sell pirate supplies . " This is the pirate supply store . You see , this is sort of a sketch I did on a napkin . A great carpenter built all this stuff and you see , we made it look sort of pirate supply-like . Here you see planks sold by the foot and we have supplies to combat scurvy ; we have the peg legs there , that are all handmade and fitted to you ; up at the top you see the eyepatch display , which is the black column there for everyday use , your regular eyepatch , and then you h
 ave the pastel and other colors for stepping out at night -- special occasions , bar mitzvahs and whatever . So we opened this place . And this is a vat that we fill with treasures that students dig in : this is replacement eyes in case you lose one ; these are some signs that we have all over the place : " Practical Joking with Pirates . " While you 're reading the sign , we pull a rope behind the counter and eight mop heads drop on your head . That was just my one thing -- I said we had to have something that drops on people 's heads . It became mop heads . And this is the fish theater , which is just a saltwater tank with three seats , and then right behind it we set up this space . Which was the tutoring center . So right there is the tutoring center , and then behind the curtain are the McSweeney 's offices , where all of us would be working on the magazine and book editing and like that . The kids would come in -- or we thought they would come in . I should back up . We set th
 e place up , we opened up , we spent months and months renovating this place . We had tables , chairs , computers , everything . I went to a dot-com auction at a Holiday Inn in Palo Alto and I bought 11 G4s with a stroke of a paddle . Anyway , we bought 'em , we set everything up and then we waited . It was started with about 12 of my friends , people that I had known for years that were writers in the neighborhood . And we sat . And at 2:30 we put a sandwich board out on the front sidewalk and it just said , " Free Tutoring for Your English-Related and Writing-Related Needs -- Just Come In , It 's All Free . " And we thought , " Oh , they 're going to storm the gates , they 're gonna love it . " And they did n't . And so we waited , we sat at the tables , we waited and waited . And everybody was becoming very discouraged because it was weeks and weeks that we waited , really , where nobody came in . And then somebody alerted us to the fact that maybe there was a trust gap , because
  we were operating behind a pirate supply store . We never put it together , you know ? And so then , around that time , I persuaded a woman named Nineveh Caligari , a longtime San Francisco educator -- she was teaching in Mexico City , she had all the experience necessary , knew everything about education , was connected with all the teachers and community members in the neighborhood . I convinced her to move up from Mexico City where she was teaching ; she took over as executive director . Immediately , she made the inroads with the teachers and the parents and the students and everything , and so suddenly it was actually full every day . And what we were trying to offer every day was one-on-one attention . The goal was to have a one-to-one ratio with every one of these students . You know , it 's been proven that 35 to 40 hours a year with one-on-one attention , a student can get one grade level higher . And so most of these students , English is not spoken in the home . They com
 e there , many times their parents -- you ca n't see it , but there 's a church pew that I bought in a Berkeley auction right there -- the parents will sometimes watch while their kids are being tutored . So that was the basis of it , was one-on-one attention . And we found ourselves full every day with kids . If you 're on Valencia Street within those few blocks at around 2:00 , 2:30 , you will get run over , often , by the kids and their big backpacks , or whatever , actually running to this space . Which is very strange , because it 's school , in a way . But there was something psychological happening there that was just a little bit different . And the other thing was , there was no stigma . Kids were n't going into the " Center-for-Kids-That-Need-More-Help " or something like that . It was 826 Valencia . First of all , it was a pirate supply store , which is insane . And then secondly , there 's a publishing company in the back . And so our interns were actually working at the
  same tables very often , and shoulder-to-shoulder , computer next to computer with the students . And so it became a tutoring center -- publishing center , is what we called it -- and a writing center . They go in , and they might be working with a high school student actually working on a novel -- because we had very gifted kids , too . So there 's no stigma . They 're all working next to each other . It 's all a creative endeavor . They 're seeing adults . They 're modeling their behavior . These adults , they 're working in their field . They can lean over , ask a question of one of these adults and it all sort of feeds on each other . There 's a lot of cross-pollination . The only problem , especially for the adults working at McSweeney 's who had n't necessarily bought into all of this when they signed up , was that there was just the one bathroom . With like 60 kids a day , this is a problem . But you know , there 's something about the kids finishing their homework in a give
 n day , working one-on-one , getting all this attention -- they go home , they 're finished . They do n't stall , they do n't do their homework in front of the TV . They 're allowed to go home at 5:30 , enjoy their family , enjoy other hobbies , get outside , play . And that makes a happy family . A bunch of happy families in a neighborhood is a happy community . A bunch of happy communities tied together is a happy city and a happy world . So the key to it all is homework ! There you have it , you know -- one-on-one attention . So we started off with about 12 volunteers , and then we had about 50. And then a couple hundred . And we now have 1,400 volunteers on our roster . And we make it incredibly easy to volunteer . The key thing is , even if you only have a couple of hours a month , those two hours shoulder-to-shoulder , next to one student , concentrated attention , shining this beam of light on their work , on their thoughts and their self-expression , is going to be absolutel
 y transformative , because so many of the students have not had that ever before . So we said , " Even if you have two hours one Sunday every six months , it does n't matter . That 's going to be enough . " So that 's partly why the tutor corps grew so fast . Then we said , " Well , what are we going to do with the space during the day , because it has to be used before 2:30 ? " So we started bringing in classes during the day . So every day , there 's a field trip where they together create a book ; you can see it being typed up above . This is one of the classes getting way too excited about writing . You just point a camera at a class , and it always looks like this . So this is one of the books that they do . Notice the title of the book , " The Book That Was Never Checked Out : Titanic . " And the first line of that book is , " Once there was a book named Cindy that was about the Titanic . " So , meanwhile , there 's an adult in the back typing this up , taking it completely se
 riously , which blows their mind . So then we still had more tutors to use . This is a shot of just some of the tutors during one of the events . The teachers that we work with -- and everything is different to teachers -- they tell us what to do . We went in there thinking , " We 're ultimately , completely malleable . You 're going to tell us . The neighborhood 's going to tell us , the parents are going to tell us . The teachers are going to tell us how we 're most useful . " So then they said , " Why do n't you come into the schools ? Because what about the students that would n't come to you , necessarily , who do n't have really active parents that are bringing them , or are n't close enough ? " So then we started saying , " Well , we 've got 1,400 people on our tutor roster . Let 's just put out the word . " A teacher will say , " I need 12 tutors for the next five Sundays . We 're working on our college essays . Send them in . " So we put that out on the wire : 1,400 tutors 
 . Whoever can make it signs up . They go in about a half an hour before the class . The teacher tells them what to do , how to do it , what their training is , what their project is so far . They work under the teacher 's guide , and it 's all in one big room . And that 's actually the brunt of what we do is , people going straight from their workplace , straight from home , straight into the classroom and working directly with the students . So then we 're able to work with thousands and thousands more students . Then another school said , " Well , what if we just give you a classroom and you can staff it all day ? " So this is the Everett Middle School Writers ' Room , where we decorated it in buccaneer style . It 's right off the library . And there we serve all 529 kids in this middle school . This is their newspaper , the " Straight-Up News , " that has an ongoing column from Mayor Gavin Newsom in both languages -- English and Spanish . So then one day Isabel Allende wrote to u
 s and said , " Hey , why do n't you assign a book with high school students ? I want them to write about how to achieve peace in a violent world . " And so we went into Thurgood Marshall High School , which is a school that we had worked with on some other things , and we gave that assignment to the students . And we said , " Isabel Allende is going to read all your essays at the end . She 's going to publish them in a book . She 's going to sponsor the printing of this book in paperback form . It 's going to be available in all the bookstores in the Bay Area and throughout the world , on Amazon and you name it . " And so these kids worked harder than they 've ever worked on anything in their lives , because there was that outside audience , there was Isabel Allende on the other end . I think we had about 170 tutors that worked on this book with them and so this worked out incredibly well . We had a big party at the end . This is a book that you can find anywhere . So that led to a 
 series of these . You can see Amy Tan sponsored the next one , " I Might Get Somewhere . " And this became an ongoing thing . More and more books . Now we 're sort of addicted to the book thing . The kids will work harder than they 've ever worked in their life if they know it 's going to be permanent , know it 's going to be on a shelf , know that nobody can diminish what they 've thought and said , that we 've honored their words , honored their thoughts with hundreds of hours of five drafts , six drafts -- all this attention that we give to their thoughts . And once they achieve that level , once they 've written at that level , they can never go back . It 's absolutely transformative . And so then they 're all sold in the store . This is near the planks . We sell all the student books . Where else would you put them , right ? So we sell 'em , and then something weird had been happening with the stores . The store , actually -- even though we started out as just a gag -- the stor
 e actually made money . So it was paying the rent . And maybe this is just a San Francisco thing -- I do n't know , I do n't want to judge . But people would come in -- and this was before the pirate movies and everything ! It was making a lot of money . Not a lot of money , but it was paying the rent , paying a full-time staff member there . There 's the ocean maps you can see on the left . And it became a gateway to the community , People would come in and say , " What the -- ? What is this ? " I do n't want to swear on the web . Is that a rule ? I do n't know . They would say , " What is this ? " And people would come in and learn more about it . And then right beyond -- there 's usually a little chain there -- right beyond , they would see the kids being tutored . This is a field trip going on . And so they would be shopping , and they might be more likely to buy some lard , or millet for their parrot , or , you know , a hook , or hook protector for nighttime , all of these thin
 gs we sell . So the store actually did really well . But it brought in so many people : teachers , donors , volunteers , everybody . Because it was street level . It was open to the public . It was n't a non-profit buried , you know , on the 30th floor of some building downtown . It was right in the neighborhood that it was serving , and it was open all the time to the public . So , it became this sort of weird , happy accident . So all the people I used to know in Brooklyn , they said , " Well , why do n't we have a place like that here ? " And a lot of them had been former educators or would-be educators , so they combined with a lot of local designers , local writers , and they just took the idea independently and they did their own thing . They did n't want to sell pirate supplies ; they did n't think that that was going to work there . So , knowing the crime-fighting community in New York , they opened the Brooklyn Superhero Supply Company . This is Sam Potts ' great design tha
 t did this . And this was to make it look sort of like one of those keysmith 's shops that has to have every service they 've ever offered , you know , all over there . So they opened this place . Inside it 's like a Costco for superheroes -- all the supplies in kind of basic form . These are all handmade . These are all sort of repurposed other products , or whatever . All the packaging is done by Sam Potts . So then you have the villain containment unit , where kids put their parents . You have the office . This is a little vault -- you have put your product in there , it goes up an electric lift and then the guy behind the counter tells you that you have to recite the vow of heroism , which you do if you want to buy anything . And it limits , really , their sales . Personally , I think it 's a problem . Because they have to do it hand on heart and everything . These are some of the products . These are all handmade . This is a secret identity kit . If you want to take on the iden
 tity of Sharon Boone , one American female marketing executive from Hoboken , New Jersey . It 's a full dossier on everything you would need to know about Sharon Boone . So , this is the capery where you get fitted for your cape , and then you walk up these three steel-graded steps and then we turn on three hydraulic fans from every side and then you can see the cape in action . There 's nothing worse than , you know , getting up there and the cape is bunching up or something like that . So then , the secret door -- this is one of the shelves you do n't see when you walk in but it slowly opens . You can see it there in the middle next to all the grappling hooks . It opens and then this is the tutoring center in the back . So you can see the full effect ! But this is -- I just want to emphasize -- locally funded , locally built . All the designers , all of the builders , everybody was local , all the time was pro-bono . I just came and visited and said , " Yes , you guys are doing gr
 eat , " or whatever . That was it . You can see the time in all five boroughs of New York in the back . So this is the space during tutoring hours . It 's very busy . Same principles : one-on-one attention , complete devotion to the students ' work and a boundless optimism and sort of a possibility of creativity and ideas . And this switch is flicked in their heads when they walk through those 18 feet of this bizarre store , right ? So it 's school , but it 's not school . It 's clearly not school , even though they 're working shoulder-to-shoulder on tables , pencils and papers , whatever . This is one of the students , Khaled Hamdan . You can read this quote . Addicted to video games and TV . Could n't concentrate at home . Came in . Got this concentrated attention . And he could n't escape it . So soon enough , he was writing . He would finish his homework early -- got really addicted to finishing his homework early . It 's an addictive thing to sort of be done with it and to hav
 e it checked and to know he 's going to achieve the next thing and be prepared for school the next day . So he got hooked on that , and then he started doing other things . He 's now been published in five books . He co-wrote a mockumentary about failed superheroes called " Super-Has-Beens . " He wrote a series on " Penguin Balboa , " which is a fighting -- a boxing -- penguin . And then he read aloud just a few weeks ago to 500 people at Symphony Space , at a benefit for 826 New York . So he 's there every day . He 's evangelical about it . He brings his cousins in now . There 's four family members that come in every day . So , I 'll go through really quickly . This is L. A. , The Echo Park Time Travel Mart : " Whenever you are , we 're already then . " This is sort of a 7-11 for time travelers . So you see everything : it 's exactly as a 7-11 would be . Leeches . Mammoth chunks . They even have their own slurpee machine : " Out of order . Come back yesterday . " Anyway . So I 'm 
 going to jump ahead . These are spaces that are only affiliated with us , doing this same thing : Word St. in Pittsfield , Massachusetts . Ink Spot in Cincinnati . Youth Speaks , San Francisco , California , which inspired us . Studio St. Louis in St. Louis . Austin Bat Cave in Austin . Fighting Words in Dublin , Ireland , started by Roddy Doyle ; this will be open in April . Now I 'm going to the TED Wish -- is that okay ? All right ; I 've got a minute . So , the TED Wish : I wish that you -- you personally and every creative individual and organization you know -- will find a way to directly engage with a public school in your area and that you 'll then tell the story of how you got involved , so that within a year we have a thousand examples -- a thousand ! -- of transformative partnerships . Profound leaps forward ! And these can be things that maybe you 're already doing . I know that so many people in this room are already doing really interesting things . I know that for a f
 act . So , tell us these stories and inspire others on the website . We created a website , I 'm going to switch to " we " and not " I " hope : We hope that the attendees of this conference will usher in a new era of participation in our public schools . We hope that you will take the lead in partnering your innovative spirit and expertise with that of innovative educators in your community . Always let the teachers lead the way . They will tell you how to be useful . I hope that you 'll step in and help out . There are a million ways . You can walk up to your local school and consult with the teachers . They 'll always tell you how to help . So -- this is with Hot Studio in San Francisco , they did this phenomenal job . This website is already up , it 's already got a bunch of stories , a lot of ideas . It 's called " Once Upon a School , " which is a great title , I think . This site will document every story , every project that comes out of this conference and around the world .
  So you go to the website ; you see a bunch of ideas you can be inspired by and then you add your own projects once you get started . Hot Studio did a great job in a very tight deadline . So , visit the site . If you have any questions , you can ask this guy , who 's our director of national programs . He 'll be on the phone . You email him , he 'll answer any question you possibly want . And he 'll get you inspired and get you going and guide you through the process so that you can affect change . And it can be fun ! That 's the point of this talk -- it need n't be sterile . It need n't be bureaucratically untenable . You can do and use the skills that you have . The schools need you . The teachers need you . Students and parents need you . They need your actual person : your physical personhood and your open minds and open ears and boundless compassion , sitting next to them , listening and nodding and asking questions for hours at a time . Some of these kids just do n't plain kno
 w how good they are : how smart and how much they have to say . You can tell them . You can shine that light on them , one human interaction at a time . So we hope you 'll join us . Thank you so much . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/150ted_don_norman_on_design_and_emotion.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/150ted_don_norman_on_design_and_emotion.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/150ted_don_norman_on_design_and_emotion.txt
new file mode 100644
index 0000000..046f19f
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/150ted_don_norman_on_design_and_emotion.txt
@@ -0,0 +1,2 @@
+
+The new me is beauty . ( Laughter ) Yeah , people used to say , you know , Norman 's OK , but if you followed what he said , everything would be usable but it would be ugly . Well , I did n't have that in mind , so ... This is neat . Thank you for setting up my display . I mean , it 's just wonderful . And I have n't the slightest idea what it does or what it 's good for , but I want it . And that 's my new life . My new life is trying to understand what beauty is about , and pretty , and emotions . The new me is all about making things kind of neat and fun . And so this is a Philippe Starck juicer , produced by Alessi . It 's so much fun I have it in my house , but I have it in the entryway , I do n't use it to make juice . ( Laughter ) In fact , I bought the gold-plated special edition and it comes with a little slip of paper that says , " Do n't use this juicer to make juice . The acid will ruin the gold-plating . " ( Laughter ) So actually , I took a carton of orange juice and I
  poured it in the glass to take this picture . ( Laughter ) But , beneath it is a wonderful knife . It 's a Global cutting knife made in Japan . First of all , look at the shape , it 's just wonderful to look at . Second of all , it 's really beautiful balanced it holds -- feels well . And third of all , it 's so sharp , it just cuts . It 's a delight to use . And so , it 's got everything , right ? It 's beautiful and it 's functional . And I can tell you stories about it , which makes it reflective , and so you 'll see I have a theory of emotion . And those are the three components . Hiroshi Ishii and his group at the MIT Media Lab took a ping-pong table and a projector above it , and on the ping-pong table they projected an image of water and fish swimming in it . And as you play ping-pong , whenever the ball hits part of the table the ripples spread out and the fish run away . But of course , then the ball hits the other side , ripples hit the -- poor fish , they ca n't find any
  peace and quiet . ( Laughter ) And , is that a good way to play ping-pong ? No. But is it fun ? Yeah ! Yeah . So -- or look at Google . If you type in , oh say , " emotion and design " you get 10 pages of results . So Google just took their logo and they spread it out . Instead of saying " You got 73,000 results . This one through 20. Next , " they just give you as many O 's as there are pages . It 's really simple and subtle . I bet a lot of you have seen it and never noticed it . That 's the subconscious mind that sort of notices it , it probably is kind of pleasant and you did n't know why . And it 's just clever . And of course , what 's especially good , if you type " design and emotion , " the first response out of those 10 pages is my website . ( Laughter ) Now , the weird thing is Google lies because if I type " design and emotion , " it says , " You do n't need the 'and . ' We do it anyway . " So , OK . So I type " design emotion " and my website was n't first again . It w
 as third . Oh well , different story . There was this wonderful review in The New York Times about the MINI Cooper automobile . It said , " You know , this is a car that has lots of faults . Buy it anyway . It 's so much fun to drive . " And if you look at the inside of the car -- I mean , I wanted to see , I rented it , this is me taking a picture while my son is driving -- and the inside of the car , the whole design is fun . It 's round , it 's neat . The controls work wonderfully . So that 's my new life , it 's all about fun . I really have the feeling that pleasant things work better , and that never made any sense to me until I finally figured out , look ... I 'm going to put a plank on the ground . So , imagine I have a plank about two feet wide and 30 feet long and I 'm going to walk on it , and you see I can walk on it without looking , I can go back and forth and I can jump up and down . No problem . Now I 'm going to put the plank 300 feet in the air and I 'm not going t
 o go near it , thank you . Intense fear paralyzes you . It actually affects the way the brain works . So , Paul Saffo , just before his talk said that he did n't really have it down until just a few days or hours before the talk , and that anxiety was really helpful in causing him to focus And that 's what fear and anxiety does It causes you to be -- what 's called depth-first processing -- to focus , not be distracted , and I could n't force myself across that . Now some people can -- circus workers , steel workers . But it really changes the way you think . And then , a psychologist , Alice Isen , did this wonderful experiment . She brought students in to solve problems So , she 'd bring people into the room , there 'd be a string hanging down here and a string hanging down here and an empty room , except a table with a bunch of crap on it -- some papers and scissors and stuff . And she 'd bring them in , and she 'd say , " This is an IQ test and it determines how well you do in l
 ife . Would you tie those two strings together ? " So they 'd take one string and they 'd pull it over here and they could n't reach the other string . Still ca n't reach it . And , basically none of them could solve it . You bring in a second group of people , and you say , " Oh , before we start , I got this box of candy , and I do n't eat candy . Would you like the box of candy ? " And turns out they liked it , and it made them happy , not very happy , but a little bit of happy . And guess what -- they solved the problem . And it turns out that when you 're anxious you squirt neural transmitters in the brain , which focuses you makes you depth-first , And when you 're happy -- what we call positive valence -- you squirt dopamine into the prefrontal lobes , which makes you a breadth-first problem solver you 're more susceptible to interruption , you do out of the box thinking . That 's what brainstorming 's about , right ? With brainstorming we make you happy , we play games , and
  we say , " No criticism , " and you get all these weird , neat ideas . But in fact , if that 's how you always were you 'd never get any work done because you 'd be working along and say , " Oh , I got a new way of doing it . " So to get work done , you 've got to set a deadline , right ? You 've got be anxious . So the brain works differently and if you 're happy , things work better because you 're more creative You get a little problem , you say , " Ah , I 'll figure it out . " No big deal . There 's something I call the visceral level of processing . Biology -- we have co-adapted through biology to like bright colors . That 's especially good that mammals and primates like fruits and bright plants , because you eat the fruit and you thereby spread the seed . There 's an amazing amount of stuff that 's built into the brain . We dislike bitter tastes , we dislike loud sounds , we dislike hot temperatures , cold temperatures . We dislike scolding voices , we dislike frowning faces
  , We like symmetrical faces , et cetera , et cetera . So that 's the visceral level and in design you can express visceral in lots of ways , like the choice of type fonts and the red for hot , exciting . Or the 1963 Jaguar . It 's actually a crummy car , falls apart all the time , but the owners love it . And it 's beautiful -- it 's in the Museum of Modern Art . A water bottle . You buy it because of the bottle , not because of the water . And when people are finished , they do n't throw it away they keep it for -- you know , it 's like the old wine bottles , you keep it for decoration or maybe fill it with water again , which proves it 's not the water . It 's all about the visceral experience . The middle level of processing is the behavioral level and that 's actually where most of our stuff gets done . Visceral is subconscious , you 're unaware of it Behavioral is subconscious , you 're unaware of it . Almost everything we do is subconscious . I 'm walking around the stage , I
  'm not attending to the control of my legs . I 'm doing a lot , most of my talk is subconscious , it 's been rehearsed and thought about a lot . Most of what we do is subconscious . Automatic behavior -- skilled behavior is subconscious , controlled by the behavioral side . And behavioral design is all about feeling in control , which includes usability , understanding , but also the feel and heft . That 's why the Global knives are so neat . They 're so nicely balanced , so sharp , you really feel you 're in control of the cutting . Or just driving a high-performance sports car over a demanding curb , again feeling that you are in complete control of the environment . Or the sensual feeling . This is a Kohler shower , a waterfall shower , and actually , all those knobs beneath are also shower heads . It will squirt you all around And you can stay in that shower for hours . And not waste water , by the way , it recirculates the same dirty water . ( Laughter ) Or this -- this is a r
 eally neat teapot I found at high tea at The Four Seasons Hotel in Chicago . It 's a Ronnefeldt tilting teapot . That 's kind of what the teapot looks like but the way you use it is you lay it on its back , and you put tea in , and then you fill it with water because water then seeps over the tea . And the tea is sitting in this stuff to the right -- the tea is to the right of this line . There 's a little ledge inside , so the tea is sitting there and the water is filling it up like that . And when the tea is ready , or almost ready , you tilt it . And that means the tea is partially covered while it completes the brewing . And when it 's finished , you put it vertically , and now the tea is -- you remember -- above this line and the water only comes to here and so it keeps the tea out And on top of that , it communicates , which is what emotion does . Emotion is all about acting , emotion is really about acting . It 's being safe in the world . Cognition is about understanding the
  world , emotion is about interpreting it saying good , bad , safe , dangerous , and getting us ready to act , which is why the muscles tense or relax . And that 's why we can tell the emotion of somebody else , because their muscles are acting , subconsciously , except that we 've evolved to make the facial muscles really rich with emotion . Well , this has emotions if you like , because it signals the waiter that , " Hey , I 'm finished . See -- upright . " And the waiter can come by and say , " Would you like more water ? " It 's kind of neat . What a wonderful design . And the third level is reflective , which is , if you like the superego , it 's a little part of the brain that has no control over what you do , no control over the -- does n't see the senses , does n't control the muscles . It looks over what 's going on . It 's that little voice in your head . that 's watching and saying , " That 's good . That 's bad . " or " Why are you doing that ? I do n't understand . " It
  's that little voice in your head that 's the seat of consciousness . Here 's a great reflective product . Owners of the Hummer have said , " You know I 've owned many cars in my life all sorts of exotic cars , but never have I had a car that attracted so much attention . " It 's about their image , it 's not about the car . But even if you want a more positive model , this is the GM car . And the reason you might buy it now is because you care about the environment And you 'll buy it to protect the environment , even though the first few cars are going to be really expensive and not perfected . But that 's reflective design as well . Or an expensive watch so you can impress people , who say " Oh gee , I did n't know you had that watch . " As opposed to this one , which is a pure behavioral watch , which probably keeps better time than the 13,000 dollar watch I just showed you . But it 's ugly . This is a clear Don Norman watch . And what 's neat is sometimes you pit one emotion ag
 ainst the other , the visceral fear of falling against the reflective state saying " It 's OK . It 's OK . It 's safe . It 's safe . " If that amusement park were rusty and falling apart , you 'd never go on the ride . So , it 's pitting one against the other . The other neat thing ( Laughter ) So Jake Cress is this furniture maker , and he makes this unbelievable set of furniture . And this is his chair with claw , and the poor little chair has lost its ball and it 's trying to get it back before anybody notices . And what 's so neat about it is how you accept that story . And that 's what 's nice about emotion . So that 's the new me . I 'm only saying positive things from now on . ( Laughter ) ( Applause ) 
\ No newline at end of file


[15/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/0syndicate_bmw-s-and-chinese-justice.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/0syndicate_bmw-s-and-chinese-justice.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/0syndicate_bmw-s-and-chinese-justice.txt
new file mode 100644
index 0000000..0d7e0a3
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/0syndicate_bmw-s-and-chinese-justice.txt
@@ -0,0 +1,2 @@
+
+BMW 's and Chinese Justice * * * * * In most places , the name " BMW " implies luxury and exclusivity . But in China , the brand has unwittingly found itself enmeshed in tales that illustrate the powerlessness of ordinary Chinese against the powerful and corrupt . The first scandal occurred last year , after a tractor filled with green onions scraped the side of a BMW in Harbin . The drivers of both the tractor and the metallic-silver luxury car were 45-year-old women , but any similarity between them ended there : the former was a peasant , the latter the wife of a wealthy businessman . After a confrontation between the two , the wealthy wife drove her BMW into the growing crowd of spectators on the roadside , killing the peasant woman and injuring 12 others . The case went to the local court , where the judge ruled it an " accidental traffic disturbance " and gave the driver a two-year suspended sentence . The judge 's ruling spurred rumors that he went easy on her because her hus
 band was related to senior provincial officials . It quickly became a story of " rich versus poor , " widely cited as an example of high-level corruption . Over the next few months , newspapers and television stations ran with the story as it snowballed into a national obsession . Alarmed by the public reaction , the Harbin magistrate ordered the case reopened , only to have the suspended sentence confirmed when the case was closed this March . The big BMW story of late 2003 resurfaced quietly , before being buried again forever . That same month , a lottery scandal threw the BMW brand into the limelight once again . Lottery officials in Shaanxi province rejected a winning ticket , calling it a fake and denying its bearer , a 17-year-old security guard named Liu Liang , the grand prize of a $58,000 BMW and 120,000 yuan ( $14,510 ) in cash . Liu became so angry about being accused of fraud and denied the car that he climbed atop a high advertising billboard and threatened to jump as 
 a show of innocence . But the story did n't end when police officers managed to talk him down . News broadcasts covered his continued insistence that he did not forge his ticket , along with the lottery center 's claims that their rejection of the ticket was legitimate . The police finally stepped in and , after a careful investigation , announced that they had found the true criminal : Yang Yongming , a private businessman whom the local lottery administration had contracted to organize ticket sales . Yang had conspired with the government officials directing the lottery , who were arrested for malfeasance , to fraudulently obtain the top prizes . In June Liu Liang finally got what he deserved - a BMW-325i sedan and a sincere apology from the lottery center . If the first scandal was a tragedy , the second was more like a farce . But both offer keys to understanding contemporary Chinese psychology . The outcry after the first BMW case was not really about the light sentence given t
 o a rich woman , but about the lack of confidence ordinary people have in China 's judicial system . In China , power , money , and connections trump the law . Even as they are becoming ever more litigious , many Chinese believe that they have no hope of securing justice against the powerful . The apathetic response of the dead peasant woman 's husband to the $10,000 in compensation he received was telling . " I do n't care about the verdict and whether it is justice or not , " he said . The most harmful consequence is the public 's loss of trust in the system . Social trust is not something you can buy with money . If an entire society believes that you cannot depend on legal rights for protection - that one must instead rely on a web of relationships with those who have power and influence - questions about whether such a society is livable or desirable will remain . Similarly , in the second BMW case , people did n't blame a corrupt businessman ; they ascribed culpability to the 
 lottery center , a government body . So instead of mistrusting one person , they grew suspicious of an institution - even of government itself . When a stubborn teenager went up against the mighty lottery authority with its army of auditors and inspectors and initial alibis , this individual , not the system , was the clear winner of the public 's admiration . Liu Liang may have been just a working-class kid , but there was wisdom in his words that there is still a " silent majority " who can affect the workings of China 's fragile society . He refused to settle privately , because he believed that if he let corrupt government officials off the hook , " they 'll keep scamming the public . " Thanks to his perseverance and the media 's investigation , the fraud was laid bare . Power corrupts everywhere , but individuals in China such as Liu have come to form a countervailing force . Even so , such marginal forces do not yet constitute a system for redressing grievances . While pop mus
 ic fans in China can listen to whatever they like , including Madonna singing " I 'm gonna shake up the system , " ordinary Chinese need courage to speak such messages aloud . As one saying goes , " There is not want of conscience in Chinese , but there is want of courage . " 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/10syndicate_france-s-fourth-moment.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/10syndicate_france-s-fourth-moment.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/10syndicate_france-s-fourth-moment.txt
new file mode 100644
index 0000000..433bbd1
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/10syndicate_france-s-fourth-moment.txt
@@ -0,0 +1,2 @@
+
+France\u2019s Fourth Moment KYIV \u2013 Since World War II\u2019s end , France has consistently risen to the challenge of restructuring Europe in times of crisis . In doing so , France became the catalyst not only for building European unity , but also for creating the prosperity that marked Europe\u2019s post-war decades \u2013 a prosperity now under threat because of the global financial and economic crisis . If we are to see a stronger Europe emerge from today\u2019s challenges , visionary French leadership is needed again . The first moment when decisive French leadership began to unify Europe came when Robert Schuman and Konrad Adenauer created the European Coal and Steel Community . By rooting the then West Germany in the political , economic , and social fabric of the West , that step heralded the start of Germany\u2019s rebirth and economic miracle . The second time that France consolidated Europe\u2019s internal architecture came in 1983 , during the debates over the stationing of American cruise 
 and Pershing missiles in Germany to counter the deployment of SS20 missiles by the Soviet Union . France\u2019s resolve to stand with Germany helped prevent Germany from drifting into a dangerous neutrality that would have shaken the European community to its core . The third moment came after the Berlin Wall fell , and it was feared that an enlarged Germany might destabilize Europe . After initially trying to delay reunification , France embraced it , in exchange for Germany\u2019s reaffirming its commitment to European unity and Franco-German leadership of the European community . As a result , Germany bound itself to the idea of \u201c ever closer union \u201d even more definitively by pledging to join the common European currency , the euro . It is now urgent that the insights that have animated Franco-German relations be applied to all of Central and Eastern Europe . Only by securing the European identity of this entire region , and by anchoring the growing Russo-German relationship in a E
 uropean context , can the European Union continue on its path of stability and prosperity . Only by reinforcing open trading and financial relations across Europe can we hope to subdue today\u2019s crisis . For it is in no European country\u2019s interest , or in the interest of the EU as a whole , that Central and Eastern Europe feel that they have been cast adrift or that Germany and Russia fixate on each other in this time of crisis . As with Germany in the 1950\u2019s , the nature of Russia\u2019s links with its immediate neighbors is turning out to be the defining factor in shaping the country\u2019s international image . Many observers regard these relations as a signal not just to the region , but to the rest of the world , of the sort of power that Russia wishes to be . The issue is partly one of \u201c internationalism \u201d itself . In contrast to Europe \u2013 with its close-knit network of multilateral organizations through which states formulate and conduct much of their foreign policies \u2013 
 Russia is not accustomed to intensively cooperative international procedures . But keeping Russia at arm\u2019s length from Europe has only strengthened the sense of isolation that many Russians feel , tempting them to define the country\u2019s interests in ways that are irreconcilable with those of Europe . It has also heightened Russia\u2019s desire to construct a special , bilateral Russo-German relationship heedless of the context of the EU . European history during the past 60 years makes clear that the most promising approach to meeting the challenge of national reconciliation and stability is not to focus on specific contingencies , but to establish procedures that encourage orderly change . A singular vision has always animated this process : animosity between neighbors must not be allowed to fester , and the rule of law must reign not just within countries , but also between them . To believe that such a vision can work for Russia , Ukraine , and Europe is not wishful thinking , but
  rests on the successful experience of France and Germany in promoting amity . Indeed , the existence or absence of a framework of cooperation often determines whether a dispute mutates into a crisis . Such cooperative frameworks seek to reconcile national independence with regional interdependence , focusing political leaders \u2019 minds on prosperity for their people rather than on gaining unilateral advantages that ultimately impoverish and unsettle everyone . The first lesson of European unity is that times of crisis must draw the continent closer together , not divide it through protectionism , competitive devaluations , and expulsions of immigrants . Similarly , the euro must not be allowed to become an Iron Curtain that consigns non-members to a high-risk zone where investors dare not venture . For Ukraine , Europe can help by embracing the free-trade agreement that we are now negotiating . Coupled with our successful membership in the World Trade Organization , Ukraine would s
 tand to benefit when world and European trade begins to recover . Europe can also consider using various stabilization funds to help our economy through the crisis that we are all enduring . I do not ask for these things out of a narrow concern for my country\u2019s health . Just as the US Federal Reserve has engaged in credit and currency swaps with Brazil , Mexico , Singapore , and South Korea and other countries to ease their access to the dollars they need , the European Central Bank must offer such swap arrangements to Europe\u2019s non-euro countries in order for trade and production processes to continue . Yes , these are dark times , and all politicians want to protect their voters . But Europe\u2019s greatest postwar leaders understood that keeping the wider view of Europe in mind is the best way to achieve this goal . As at so many times before , now \u2013 with all of our economies in peril \u2013 is a moment for decisive French leadership . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/11syndicate_fran-ois-hollande-s-wrong-idea-of-france-by-brigitte-granville.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/11syndicate_fran-ois-hollande-s-wrong-idea-of-france-by-brigitte-granville.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/11syndicate_fran-ois-hollande-s-wrong-idea-of-france-by-brigitte-granville.txt
new file mode 100644
index 0000000..6f00088
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/11syndicate_fran-ois-hollande-s-wrong-idea-of-france-by-brigitte-granville.txt
@@ -0,0 +1,2 @@
+
+Fran�ois Hollande\u2019s Wrong Idea of France LONDON \u2013 France\u2019s new president , Fran�ois Hollande , has achieved a remarkable series of political victories \u2013 at home and in Europe \u2013 since his election in May . Unfortunately , his streak of success will inevitably call forth an economic reckoning that will shock France\u2019s apparently unsuspecting citizens and doom the French elite\u2019s approach to the \u201c construction of Europe . \u201d Since winning the presidency , Hollande has won a parliamentary majority and pushed Germany toward accepting joint liability for eurozone countries \u2019 debts . But forebodings of crisis have become widespread in French business and economic circles . But the real danger \u2013 which even Hollande\u2019s sternest critics may be underestimating \u2013 is not so much his individual policy failings ( serious though they may be ) as his approach to the twin challenges posed by France\u2019s economic imbalances and the eurozone crisis . On each front separately , he
  might manage to muddle through ; together , they look likely to cement France\u2019s loss of competitiveness . Declining competitiveness is best captured in a single indicator : unit labor costs , which measure the average cost of labor per unit of output . In a monetary union , discrepancies in wage growth relative to productivity gains \u2013 that is , unit labor costs \u2013 will result in a chronic accumulation of trade surpluses or deficits . Since the euro\u2019s introduction , unit labor costs have risen dramatically faster in France than they have in Germany . According to Eurostat data published in April 2011 , the hourly labor cost in France was \u20ac34.2 , compared to \u20ac30.1 in Germany \u2013 and nearly 20 % higher than the eurozone average of \u20ac27.6 . France\u2019s current-account deficit has risen to more than 2 % of GDP , even as its economic growth has ground to a halt . The high cost of employing workers in France is due not so much to wages and benefits as it is to payroll taxes lev
 ied on employers . The entire French political class has long delighted in taxing labor to finance the country\u2019s generous welfare provisions , thus avoiding excessively high taxation of individuals \u2019 income and consumption \u2013 though that is about to come to an end as Hollande intends to slap a 75 % tax on incomes above \u20ac1 million . This is a version of the fallacy that taxing companies ( \u201c capital \u201d ) spares ordinary people ( \u201c workers \u201d ) . Of course , such taxes on firms are always passed on to households \u2013 usually through straightforward price hikes , and , in France , also through unemployment . High tax rates on labor \u2013 together with rigid regulation of hiring and firing \u2013 make employers extremely reluctant to recruit workers . As a result , France has had chronic long-term unemployment \u2013 forecast to reach 10.5 % by 2013 \u2013 for many years . Hollande\u2019s predecessor , Nicolas Sarkozy , tried to address this problem . He exempted voluntary overtime pay fro
 m employment tax and shifted some of the burden of labor taxation onto consumption ( via a hike in VAT ) . But Hollande quickly reversed both of these reforms . The repeal of the tax break on overtime reflects another economic fallacy to which French Socialist politicians are deeply attached : the \u201c lump of labor \u201d notion that underlay the most disastrous of their economic policies \u2013 the 35-hour workweek , introduced in 2000. The idea behind the policy is that demand for labor is a constant , and that this fixed number of aggregate working hours required by employers to meet final demand can be spread more evenly among workers to reduce unemployment . Such measures , designed to create jobs by freeing up work hours , are futile at best , and are often detrimental . French Socialists should recall their school physics lesson about communicating vessels : when a homogeneous liquid is poured into a set of connected containers , it settles at the same level in all of them , regard
 less of their shape and volume . Generating more \u201c liquid \u201d ( jobs ) requires not discouraging the entrepreneurs on whose activities sustainable job creation ultimately depends . The effect of fiscal and regulatory pressure on employment is to encourage French firms to invest and hire outside France . Hollande\u2019s apologists praise his gradualist and consensual approach to addressing the economy\u2019s structural distortions . They argue that his penchant for setting up consultative commissions is the best way to forge the consensus required for structural reform , whereas Sarkozy\u2019s combative style was counterproductive . Even banishing skepticism and assuming that Hollande could over time persuade his supporters to embrace competitiveness-boosting policies , the eurozone crisis is denying France the time that such gradualism requires . A simple , effective way to buy time would be to abandon the euro and restore competitiveness through a devalued national currency . But this exp
 edient is incompatible with mainstream French politicians \u2019 devotion to the \u201c European project , \u201d which amounts to a projection of French soft power ; indeed , building Europe lies at the heart of the French establishment\u2019s version of what Charles de Gaulle used to call \u201c a certain idea of France . \u201d For mainstream French politicians , renouncing the European project to buy the time required to restore competitiveness is as unthinkable as is the logical alternative : an all-out push for full European political union . This would reestablish monetary sovereignty and create a normal central bank ( like the Federal Reserve or the Bank of England ) at the European level . But it would also mean abandoning France\u2019s republic in favor of a federal European government \u2013 anathema to that \u201c certain idea of France . \u201d The combination of gradualism ( on the most generous interpretation ) in domestic economic reform and the paralyzing effect of the eurozone crisis will lead 
 to a massive shock . Remaining in a currency union with the much more competitive German economy will require wrenching and rapid reforms , for which Hollande\u2019s tepid approach will fail to prepare the complacent French . The result will be even more support than was seen in last April\u2019s presidential election for extremist political parties that reject both Europe and competitive market capitalism . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/12syndicate_hollywood-s-favorite-villains.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/12syndicate_hollywood-s-favorite-villains.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/12syndicate_hollywood-s-favorite-villains.txt
new file mode 100644
index 0000000..00bbb81
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/12syndicate_hollywood-s-favorite-villains.txt
@@ -0,0 +1,2 @@
+
+Hollywood\u2019s Favorite Villains Once upon a time , Cold War enemies , white supremacists , and evil geniuses reigned supreme as Hollywood\u2019s favorite bad guys . No more . Today , it is multinational corporations that are increasingly being cast as the �ber-villains of our globalized world . For all their subliminal paid promotions and subtle product placements , corporations are getting drubbed in the main story lines of our popular culture . This treatment goes far beyond documentaries like Michael Moore\u2019s polemical Fahrenheit 9/11 or The Corporation , an earnest if somewhat paranoid portrayal of multinational companies \u2019 role in globalization . It extends to mainstream hits like The Constant Gardener , in which the idealistic protagonists do battle with a malicious global pharmaceutical company that is bent on exploiting Africa\u2019s misery to test experimental drugs . To be sure , sociopathic corporations have populated books and films for more than a century . But corporate 
 villains , typically multinational companies , have never been so ubiquitous as today . Is it unfair ? Most corporations , after all , are merely convenient mechanisms for ensuring that scarce global capital is used at maximum efficiency , to the benefit of all . Are famously liberal Hollywood film directors spending too much time going to anti-globalization rallies ? Perhaps . But I would submit that Hollywood\u2019s misgivings , however untutored , represent only the tip of a growing iceberg of resentment against the perceived injustices of globalization . The simple truth is that corporations represent capital , and capital \u2013 in the form of factories , equipment , machines , money , and even houses \u2013 has been the single biggest winner in the modern era of globalization . Corporate profits are bursting at the seams of investors \u2019 expectations in virtually every corner of the world . Even in moribund economies like Germany and Italy , where employment security is vanishing , cor
 porations are swimming in cash . This phenomenon comes as no surprise to economists . Add two billion Indian and Chinese workers to the global labor force , and the value of other means of production \u2013 particularly capital and commodities ( for example , gold and oil ) \u2013 is bound to go up . And so it has , with capitalists everywhere gaining an ever larger share of the economic pie . ( In theory , capitalists in labor-abundant China and India could end up as losers , but in practice they , too , have benefited thanks to their governments \u2019 success in simultaneously liberalizing and globalizing . ) Many policymakers seem to be under the impression that surging profits are a purely cyclical phenomenon , as economies continue to grow out from the depths of the 2001 recession . Wait a bit , they predict , and wages will fully catch up later in the cycle . Not likely . Capital\u2019s piece of the pie has been getting bigger for more than 20 years , and the trend looks set to continue 
 . Indeed , corporations \u2019 growing share of income has been a major driver behind the long , if uneven , bull market in stocks that began in the early 1990\u2019s . At the same time , inflation-adjusted wages for rich-country unskilled workers have barely budged over the last two decades . Some of these trends also have to do with the nature of modern technological change , which seems to favor capital and skilled workers disproportionately . But , regardless of their cause , rapidly growing inequalities are a powerful force for instability everywhere , from wealthy America to rapidly growing China to reform-challenged Europe . \u201c A rising tide lifts all boats , \u201d conservatives like to say . Fine , but what happens to people , like the poor of hurricane-struck New Orleans , who don\u2019t own boats ? Growing inequality would not be such a problem if governments could simply raise taxes on the rich and strengthen subsidies to the poor . Unfortunately , any country that taxes capital to
 o aggressively will only succeed in chasing it to regions where the tax burden is lighter . In a globalized world , national governments \u2019 ability to tax potentially mobile factors of production is sharply circumscribed . The same mechanism that pours profits into the pockets of global corporations also prevents governments from claiming a larger share of the spoils . Unfortunately , the long-term trend towards ever-lower income shares for unskilled workers is likely to continue over the coming decades , as modern technology permeates the globe , and as emerging markets like China , India , Brazil , and Eastern Europe continue to integrate into global production . This is not to say that unskilled workers are actually being made worse off by globalization in absolute terms ; the vast majority are actually breaking even or better . But unskilled workers \u2019 incomes are not keeping pace with overall economic growth , and the resulting social strains are a ticking bomb . If so , then
  Hollywood\u2019s cartoon-like caricatures of evil multinational corporations may some day seize mainstream consciousness , leading to political upheavals that shatter today\u2019s social contract . That won\u2019t be good for profits , or for the poor . Governments \u2013 and corporations \u2013 must find better ways to provide equal opportunity through improved education , broader financial markets , and other channels . Otherwise , globalization\u2019s storyline may not proceed according to the script . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/13syndicate_japanese-lessons-for-china-s-currency.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/13syndicate_japanese-lessons-for-china-s-currency.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/13syndicate_japanese-lessons-for-china-s-currency.txt
new file mode 100644
index 0000000..057a055
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/13syndicate_japanese-lessons-for-china-s-currency.txt
@@ -0,0 +1,2 @@
+
+Japanese Lessons for China\u2019s Currency SHANGHAI \u2013 The question of how much China\u2019s currency should appreciate to rebalance its trade has become a global hot-button issue . But the answers have been all over the map , with some finding that the yuan is not undervalued at all , while others argue that it should appreciate against the dollar by more than 30 % . Clearly , there must be major differences in the macroeconomic models used to produce such a wide range of estimates . But the one thing about which everyone seems to agree is the theoretically and empirically unjustified assumption that an equilibrium exchange rate actually exists . The theoretical problem is simple : a country\u2019s trade balance depends on a lot more than the value of its currency in the foreign exchange markets . Interest rates , employment , aggregate demand , and technological and institutional innovation all play a role . As the economist Joan Robinson pointed out in 1947 , just about any exchange rate
  will be the equilibrium value for some combination of these other variables . The equilibrium exchange rate , she famously argued , is a chimera . Not surprisingly , the empirical evidence that trade imbalances can be resolved through exchange rate changes alone is unconvincing . In the case of China , the most useful precedent is probably that of Japan in the period from the end of the Bretton Woods fixed exchange-rate regime in August , 1971 , to the collapse of its \u201c bubble economy \u201d in 1990. During that period , the yen\u2019s value more than doubled against the dollar , rising from its original fixed rate of 360 to 144 at the end of 1989. Yet , even as Japan\u2019s exports became much more expensive in dollar terms and its imports much cheaper in yen , its trade surplus rose from $6 billion in 1971 to $80 billion in 1989. For two decades , expectations that an appreciating yen would restore external balance were repeatedly disappointed . At the time of the December 1971 Smithson
 ian Agreement , 308 yen to the dollar was supposed to do the trick . Fourteen years later , during the Plaza Accord negotiations , the Japanese argued for an eventual level of 200-210 , while some US Treasury officials thought the final target should be as high as 165-170 . At the end of the 1980\u2019s , some analysts thought rates as high as 120 might finally produce the long-sought equilibrium . Yet , as Japan entered the \u201c lost decade \u201d of the 1990\u2019s , its exports continued to grow faster than its imports . Japan\u2019s trade surplus peaked only in 1994 , at $144 billion , just a few months before the yen\u2019s April 1995 all-time high of 79.75 . In retrospect , it is easy to see why none of these supposed equilibrium exchange rates delivered external balance . As the yen appreciated , Japan responded not by exporting less but by improving productivity and quality control through plant and equipment investment and innovations in factory management , making possible rapid growth in
  exports of high-value-added products . Exchange-rate equilibrium calculations from the 1970\u2019s and 1980\u2019s , which could only have been based on the export sector\u2019s contemporary structure , naturally would have little relevance subsequently . The same is true of calculations at the beginning of the 1990\u2019s , which would have forecast Japanese import growth based on extrapolations from the high GDP growth rates of the previous 40 years rather than on the decade of stagnation that ensued . In China , changes in the export sector\u2019s structure similar to those observed in Japan are now taking place . These changes are likely to make today\u2019s attempts to find an equilibrium yuan-dollar exchange rate seem just as chimerical in hindsight as previous calculations of the yen-dollar equilibrium rate . For the 30 years since the beginning of China\u2019s economic reforms , Chinese industry has achieved impressive efficiency gains by adopting new technologies and realizing economies of sca
 le , leading to a huge expansion in locally made products suitable for export . While an appreciating currency might eventually drive labor-intensive manufacturers out of business , if Japan\u2019s economic history is any guide , they are likely to be replaced by producers of things like ships , machine tools , semiconductors , and doubtless new products yet to be invented . The equilibrium yuan-dollar rate is a chimera not because China\u2019s trade could never be balanced , but because the exchange rate alone does not determine equilibrium . The structure of the entire economy matters , too . As this is constantly evolving in unpredictable ways , there is no reason to expect that the assumptions underlying any particular macroeconomic model will ever remain valid long enough for its steady-state solution to be achieved in practice . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/14syndicate_kennedy--khrushchev--the-cuban-missile-crisis-and-nuclear-deterrence-from-cuba-to-iran-by-joseph-s--nye.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/14syndicate_kennedy--khrushchev--the-cuban-missile-crisis-and-nuclear-deterrence-from-cuba-to-iran-by-joseph-s--nye.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/14syndicate_kennedy--khrushchev--the-cuban-missile-crisis-and-nuclear-deterrence-from-cuba-to-iran-by-joseph-s--nye.txt
new file mode 100644
index 0000000..47d87dc
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/14syndicate_kennedy--khrushchev--the-cuban-missile-crisis-and-nuclear-deterrence-from-cuba-to-iran-by-joseph-s--nye.txt
@@ -0,0 +1,2 @@
+
+The Cuban Missile Crisis at 50 CAMBRIDGE \u2013 This month marks the 50th anniversary of the Cuban missile crisis \u2013 those 13 days in October 1962 that were probably the closest the world has come to a major nuclear war . President John F. Kennedy had publicly warned the Soviet Union [ 178]not to introduce offensive missiles into Cuba . But Soviet leader Nikita Khrushchev decided to cross Kennedy\u2019s red line surreptitiously and confront the Americans with a fait accompli . When an American surveillance plane discovered the missiles , the crisis erupted . Some of Kennedy\u2019s advisers urged an air strike and invasion to destroy the missiles . Kennedy mobilized troops , but also bought time by [ 179]announcing a naval blockade of Cuba . The crisis subsided when Soviet ships carrying additional missiles turned back , and Khrushchev agreed to remove the existing missiles from the island . As then US Secretary of State Dean Rusk put it : \u201c We were eyeball to eyeball , and I think the oth
 er fellow just blinked . \u201d At first glance , this was a rational and predictable outcome . The United States had a 17-to-1 advantage in nuclear weaponry . The Soviets were simply outgunned . And yet the US did not preemptively attack Soviet missile sites , which were relatively vulnerable , because the risk that even one or two of the Soviet missiles would be fired at an American city was enough to deter a first strike . In addition , both Kennedy and Khrushchev feared that rational strategies and careful calculation might spin out of control . Khrushchev offered a vivid metaphor in one of his letters to Kennedy : [ 180]\u201cWe and you ought not now to pull on the ends of the rope in which you have tied the knot of war . \u201d In 1987 , I was part of a group of scholars that met at Harvard University with Kennedy\u2019s surviving advisers to study the crisis . Robert McNamara , Kennedy\u2019s secretary of defense , said he became more cautious as the crisis unfolded . At the time , he thoug
 ht that the probability of nuclear war resulting from the crisis might have been one in 50 ( though he rated the risk much higher after he learned in the 1990\u2019s that the Soviets had already delivered nuclear weapons to Cuba ) . Douglas Dillon , Kennedy\u2019s treasury secretary , said he thought that the risk of nuclear war had been about zero . He did not see how the situation could possibly have escalated to nuclear war , and thus had been willing to push the Soviets harder and to take more risks than McNamara was . General Maxwell Taylor , the chairman of the Joint Chiefs of Staff , also believed that the risk of nuclear war was low , and he complained that the US let the Soviet Union off too easily . He felt that the Americans should have removed the Castro regime . But the risks of losing control of the situation weighed heavily on Kennedy , too , which is why he took a more prudent position than some of his advisers would have liked . The moral of the story is that a little nuc
 lear deterrence goes a long way . Nonetheless , there are still ambiguities about the missile crisis that make it difficult to attribute the outcome entirely to the nuclear component . The public consensus was that the US won . But how much the US won , and why it won , is hard to determine . There are at least two possible explanations of the outcome , in addition to Soviet acquiescence to America\u2019s superior nuclear firepower . One focuses on the importance of the two superpowers \u2019 relative stakes in the crisis : the US not only had a greater stake in neighboring Cuba than the Soviets did , but could also bring conventional forces to bear . The naval blockade and the possibility of a US invasion strengthened the credibility of American deterrence , placing the psychological burden on the Soviets . The other explanation questions the very premise that the Cuban missile crisis was an outright US victory . The Americans had three options : a \u201c shoot-out \u201d ( bomb the missile si
 tes ) ; a \u201c squeeze out \u201d ( blockade Cuba to convince the Soviets to withdraw the missiles ) ; and a \u201c buyout \u201d ( give the Soviets something they want ) . For a long time , the participants said little about the buyout aspects of the solution . But subsequent evidence suggests that a quiet US promise to remove its obsolete missiles from Turkey and Italy was probably more important than was thought at the time ( the US also gave a public assurance that it would not invade Cuba ) . We can conclude that nuclear deterrence mattered in the crisis , and that the nuclear dimension certainly figured in Kennedy\u2019s thinking . But it was not the ratio of nuclear weapons that mattered so much as the fear that even a few nuclear weapons would wreak intolerable devastation . How real were these risks ? On October 27 , 1962 , just after Soviet forces in Cuba shot down a US surveillance plane ( killing the pilot ) , a similar plane taking routine air samples near Alaska inadvertently viola
 ted Soviet air space in Siberia . Fortunately , it was not shot down . But , even more serious , unbeknownst to the Americans , Soviet forces in Cuba had been instructed to repel a US invasion , and had been authorized to use their tactical nuclear weapons to do so . It is hard to imagine that such a nuclear attack would have remained merely tactical . Kenneth Waltz , an American scholar , recently published an article entitled [ 181]\u201cWhy Iran Should Get the Bomb . \u201d In a rational , predictable world , such an outcome might produce stability . In the real world , the Cuban missile crisis suggests that it might not . As McNamara put it , \u201c We lucked out . \u201d 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/15syndicate_merkel-in-china-by-sanjaya-baru.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/15syndicate_merkel-in-china-by-sanjaya-baru.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/15syndicate_merkel-in-china-by-sanjaya-baru.txt
new file mode 100644
index 0000000..441c20d
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/15syndicate_merkel-in-china-by-sanjaya-baru.txt
@@ -0,0 +1,2 @@
+
+Merkel in China NEW DELHI \u2013 German Chancellor Angela Merkel\u2019s second visit to China in a year comes against the backdrop of dire forecasts of a difficult September for the eurozone . Mindful of such concerns and persistent pessimism in global financial markets , Merkel is now taking bold political initiatives at home and overseas . Indeed , her China trip should be seen as an effort to assert leadership across the eurozone . At home , Merkel recently sent out a clear message to her critics that Germany must pay a price for eurozone leadership . She cautioned her colleagues against loose talk about a \u201c Grexit \u201d \u2013 Greece\u2019s exit from the eurozone \u2013 and assured visiting Greek Prime Minister Antonis Samaras that Germany remained committed to his country\u2019s membership of the eurozone . While it required courage to take such a tough stance , doing so helped to bolster her position at home and throughout the eurozone . There is now no doubt that Merkel is willing to commit Ge
 rmany to the cause of preserving both the European Union and the eurozone , and that she will work to achieve that goal . If she succeeds , she will emerge as the first great European leader of the twenty-first century . This stance suggests that Merkel appreciates the essence of the argument that Fran�ois Heisbourg , the chairman of the council of the International Institute for Strategic Studies ( IISS ) , advanced in a [ 179]recent essay : a federal arrangement does not fall apart because of problems at the periphery but because of \u201c failure at the heart of the system . \u201d Merkel has come to terms with Germany\u2019s position \u2013 and that it must act to preserve the unity of the whole . The eurozone ( and probably the EU ) cannot be saved in Greece or Finland if it cannot be saved in Germany . Committing Germany to that objective is precisely what Merkel has defined as her political goal for the rest of her term in office . Within hours of declaring her leadership at home , Merk
 el announced her visit to Beijing . The timing was highly significant : Merkel goes to China , the emerging global power , after declaring her own commitment to strengthening the eurozone , if not the EU as a whole , as a credible pole of an emerging multipolar world . In recognition of Germany\u2019s growing significance , Chinese Prime Minister Wen Jiabao welcomed a delegation of German business leaders to Beijing this May with the words : \u201c Stronger Chinese-German cooperation is good for the two countries , good for China-EU relations , and good for world prosperity and stability . The strategic dimension of Chinese-German relations , therefore , can only be strengthened , not weakened . \u201d There are several dimensions to the bilateral relationship . Germany needs both China\u2019s markets and the funds that its government can deploy to purchase German and European bonds . It also has more than 7,500 enterprises operating in China , with gross investment totaling $18.5 billion . Mor
 eover , Germany has sold $15 billion worth of technology to China , and bilateral trade hit a high of $169 billion in 2011 , accounting for 30 % of total China-EU trade . The two countries have set a bilateral trade target of $280 billion for 2015. Trade , however , is not the only immediate concern . Far more important , especially for Germany , is to get China to invest in and hold its bonds . In mid-August , at the First IISS Oberoi Lecture in Mumbai , Klaus Regling , the CEO of the European Financial Stability Facility , underscored the importance of Chinese demand for EFSF bonds and China\u2019s role in stabilizing the eurozone . Regling also revealed that there is now increased and frequent coordination between monetary authorities in the United States , Germany , and China , drawing attention to the fact that the \u201c old Triad \u201d of the dollar , euro , and yen may now have been replaced by a \u201c new Triad \u201d of the dollar , euro , and Chinese renminbi . While Regling spoke of 
 an emerging \u201c multipolar monetary system , \u201d his remarks clearly indicated the functioning of a \u201c tripolar \u201d system . Lying at two ends of that monetary triangle , the eurozone and China have acquired a geopolitical stake in helping each other . [ 180]Hans Kundnani and Jonas Parello-Plesner of the European Council on Foreign Relations view the China-Germany relationship as one that will shape the overall China-EU relationship . Like many strategic analysts , they worry whether Germany , in pursuit of purely short-term economic interests , might forsake long-term strategic interests and concerns about human rights , environmental problems , press freedoms , and other political and geopolitical issues . It is significant , therefore , that German political parties have emphasized the need for Merkel to raise such matters in her talks with Chinese leaders . However , Merkel\u2019s focus may well remain on trade , investment , and currency flows . After all , unless she can turn ar
 ound the eurozone , her rising profile at home and in Europe could easily wither . What this means is that to secure German leadership of Europe , and her own leadership of Germany , Merkel has to \u201c walk on two legs , \u201d so to speak . She needs to balance both geo-economic and geopolitical factors \u2013 both interests and values \u2013 in advancing Germany\u2019s relations with China . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/16syndicate_ms--europe-or-frau-germania-.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/16syndicate_ms--europe-or-frau-germania-.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/16syndicate_ms--europe-or-frau-germania-.txt
new file mode 100644
index 0000000..ace7b1a
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/16syndicate_ms--europe-or-frau-germania-.txt
@@ -0,0 +1,2 @@
+
+Ms. Europe or Frau Germania ? BERLIN \u2013 Just what is the matter with Angela Merkel ? Only a short while ago , she was celebrated as \u201c Ms. Europe \u201d ; now , she increasingly gives the impression of being Frau Germania . Instead of providing resolute leadership in the global financial and economic crisis , the European Union\u2019s largest economy is withdrawing into its shell . Germany has always been the motor of European integration , in accordance with its political and economic interests . Every post-World War II government has always been willing to harness Germany\u2019s financial strength to further that aim , i. e. , to foot the bill for Europe . The motto was simple : Germany gives and profits in turn . Should Germany spurn the first part of this formula , the European project would suffer serious damage \u2013 and so will German national interests . Yet this is the direction in which Chancellor Merkel seems to be heading . Indeed , in the wake of the Greek crisis , Merkel spoke 
 publicly of the need to be able to exclude eurozone members that do not play by the rules . Does she seriously believe that the euro and the EU would survive such punitive action ? Instead of ideas for strengthening the solidarity and stability of the eurozone , absurd proposals are floated in Berlin . The idea that Germany would show financial solidarity with weakened eurozone members only if they accept draconian stabilization measures is similarly unrealistic . Such measures would only exacerbate deflation in these countries , which are the most important markets for German exports . What is essential is the right combination of stability and financial aid , even if the latter is unpopular with the German public . To be sure , eurozone member states also needed the courage to respond to the crisis of confidence confronting the common currency with concerted action for tighter control of national budgets and improved cooperation . As a first step , however , Greece must be given h
 elp , through the financial solidarity of EU governments . On the other hand , foreign criticism of Germany for running a current-account surplus similar to China\u2019s is unfounded . Such criticism disregards two important differences : first , unlike China , Germany , as a member of the eurozone , cannot undervalue its currency . Second , German \u201c exports \u201d go mainly to the EU . To demand that Germany should stop increasing its competitiveness , or even reduce it , is simply bizarre . From 1990 to 2005 , Germany struggled with high unemployment , slow growth , and low competitiveness , and Europe\u2019s economy suffered as a result . Recovery was difficult to achieve , but the Common Market has profited from Germany\u2019s renewed competitiveness . The crucial point is not the German economy\u2019s renewed strength , but whether Germany will use this strength in Europe\u2019s interest or its own . Unfortunately , Merkel seems to have decided on the second option , because it entails fewer d
 omestic political risks . The responsibility for the current conflict within the EU rests with eurozone governments , but primarily with Germany and France , the zone\u2019s two strongest economies . Rather than lead , the Franco-German couple is constantly , and publicly , at each other\u2019s throats . While today\u2019s quarrel is about who should pay for restructuring Greece , the real issue is the latent distrust between the two partners , which carries the danger of a permanent estrangement . From the German point of view , France only wants to solve its national budget and debt problems at Germany\u2019s expense , simultaneously weakening Germany\u2019s competitiveness . The French government , on the other hand , fears that the Germans \u2019 commitment to eurozone stability is a ploy aimed at pushing France into a corner and leaving it behind economically . Since the global crisis erupted in the fall of 2008 , both Merkel and French President Nicolas Sarkozy have been faced with threats that
  their political majorities might disappear if they put aside their national interests in favor of a European compromise . France has no chance of meeting Germany\u2019s stabilization objectives unless Sarkozy wants to forget about re-election . Merkel would cause outrage among her conservative voters ( as well as court defeat at the German Constitutional Court in Karlsruhe ) should she agree to a more free-spending policy , including any direct financial assistance for Greece . Soon , Helmut Kohl , Europe\u2019s Honorary Citizen and Germany\u2019s Chancellor of Reunification , will be celebrating his 80th birthday . As is usual on such occasions , there will be a lot of lofty speeches about Europe . But , in view of the current situation , we can safely ignore these . What Europe needs in this serious crisis are statesmen and women of Kohl\u2019s caliber , not domestic politicians ! As the big economic and political winner of the eurozone , Germany , in particular , cannot let a serious crisis
  of confidence threaten the European project , because nearly two-thirds of its exports go to the EU . Ever since the collapse of Lehman Brothers in September 2008 , it has been clear that the global crisis would challenge the EU and the euro , because Europe lacks a common government and fiscal policy . Coordination within the eurozone \u2013 above all between its most important economies , France and Germany \u2013 is therefore all the more important . It is in Berlin and Paris , first and foremost , that the crisis-management strategy for the eurozone must be decided . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/17syndicate_reshaping-china-s-government-services-supply-chain-by-andrew-sheng-and-geng-xiao.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/17syndicate_reshaping-china-s-government-services-supply-chain-by-andrew-sheng-and-geng-xiao.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/17syndicate_reshaping-china-s-government-services-supply-chain-by-andrew-sheng-and-geng-xiao.txt
new file mode 100644
index 0000000..6400c93
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/17syndicate_reshaping-china-s-government-services-supply-chain-by-andrew-sheng-and-geng-xiao.txt
@@ -0,0 +1,2 @@
+
+Reshaping China\u2019s Government-Services Supply Chain HONG KONG \u2013 A supply chain links producers and consumers through a complex web of outsourcing contracts , with market leaders in any product category orchestrating activities to produce components profitably along its entire length . For example , an iPad is designed in California \u2013 with chips from Japan and parts from South Korea , Taiwan , and elsewhere \u2013 and finally assembled in China for global distribution . But the ecology of supply chains is not as straightforward as this depiction suggests . Most studies of supply chains examine their operations , but take for granted governments \u2019 critical enabling role . Because the non-delivery of government services would inhibit the proper functioning of business supply chains , understanding how the government-services supply chain works is vital . For example , the Chinese economy\u2019s transformation was enabled by the synchronized delivery of government services to support t
 he logistics , finance , and manufacturing supply chains . This was a complex task that involved different levels of the Chinese government and many state agencies and ministries . A supply chain is not only a network for production , but also a live feedback mechanism , continually adjusting itself to ensure that production is coordinated and aligned efficiently to meet changes in global consumers \u2019 demand , tastes , and preferences . Technology has enabled faster , more efficient \u201c just-in-time \u201d delivery , taking full advantage of specialization and knowledge-sharing on a global scale . As Apple has discovered , the winner in orchestrating a supply chain emerges with the lowest global costs and the largest market share . The iPad could not be produced at such high speed and low cost without the \u201c made-in-the-world \u201d supply chain based in China . In addition to the macro and micro aspects of economics , understanding supply chains in private and public goods and services
  in China requires [ 178]mezo ( institutional ) and meta ( system-wide ) analysis . When China initiated its economic reforms in 1979 , it inherited a centrally planned economy that lacked the institutional infrastructure for markets . Recognizing the need for systemic change , China allowed local governments in special economic zones and cities to experiment with modern legal , administrative , and logistical practices for export industries , including investments in utilities and transport . Intense competition among local governments for foreign investment led to dramatic improvements in the business environment , featuring economic incentives in areas like land , labor , and taxation , as well as speedy issuance of permits and approvals . City leaders were given responsibility for mobilizing local resources to generate GDP and employment , and they were ( and continue to be ) rewarded with promotion for good performance . The result was considerable innovation and institutionali
 zation of local government services to support market activities , including outsourcing of expertise in infrastructure project design , administration , and operations to private and foreign consulting and design companies . To support China\u2019s participation in global manufacturing supply chains , many local governments sold and dismantled their state-owned enterprises ( SOEs ) , enabling many new private firms to provide the services needed for an export-oriented , market-based economy . At the national level , the consolidation of SOEs and banks , and the modernization of their corporate governance via public listing on stock exchanges , enabled improved efficiency in regulated utilities , hard infrastructure , and resource sectors , complementing liberalization and market growth . The Chinese government-services supply chain also benefited substantially from a meritocratic human-resources tradition . Officials with substantial and successful experience in local governments , mi
 nistries , or SOEs were deliberately promoted and cross-posted to less-developed regions to spread know-how , technology , and best practices and processes . Indeed , China\u2019s economic success reflects the depth of administrative and market experience embedded in the Chinese bureaucracy . Chinese mayors are CEOs of their local economy , responsible not only for market development , but also for social stability . The critical mechanism for orchestrating and implementing the complex web of contracts embodied in China\u2019s government-services supply chain is the Five-Year Plan , which foresees vertical and horizontal integration of almost all Party and administrative agencies . The FYP uses broad objectives and targets for social and economic development , formed after extensive internal and public consultations . These mandates are translated by sub-national officials into projects and work plans , such as targets for reducing energy use per unit of GDP in order to address resource c
 onstraints and concern about climate change . China\u2019s success in developing from scratch a modern government-services delivery system explains why many foreign investors find it much easier to deal with Chinese governments than those in other developing countries . The 12th FYP aims to shift China from an export-driven growth model toward a balanced economy that relies on domestic demand , while simultaneously addressing industrial transformation , social inequities , and environmental degradation . This implies more complex contracts that go beyond promoting markets , GDP growth , and employment to ensure inclusive , equitable , and high-quality government-services delivery . Implementation of these evolving social goals through local government agencies by specific officials is a daunting task that requires profound changes in roles and performance metrics . No one doubts that Chinese local governments play a much more active and intrusive role than their counterparts in the Wes
 t , which implies an additional complicating factor . Local governments now face not only growing demands from the emerging middle class for greater transparency , competition , fairness , and access to opportunities , but also deepening conflicts between local interests and global rules . Orchestrating a complex government-services supply chain in a substantially open continental economy with 1.3 billion people and five levels of government is difficult enough using a simple GDP growth objective . Adapting the governance metric in a country of China\u2019s size to an economy that is green , inclusive , and equitable presents a novel challenge in human history . The only precedent for such an achievement is China itself . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/18syndicate_russia-s-ukrainian-path-to-the-future.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/18syndicate_russia-s-ukrainian-path-to-the-future.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/18syndicate_russia-s-ukrainian-path-to-the-future.txt
new file mode 100644
index 0000000..bfe4ec3
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/18syndicate_russia-s-ukrainian-path-to-the-future.txt
@@ -0,0 +1,2 @@
+
+Russia\u2019s Ukrainian Path to the Future MOSCOW \u2013 Russia and the West are losing each other yet again . The magnetic attraction and repulsion between the two has been going on for centuries . Indeed , historians have counted as many as 25 such cycles since the reign of Tsar Ivan III . In the past , however , Russia\u2019s sharp anti-Western turns were reversed \u2013 usually out of simple necessity \u2013 after relations reached rock bottom . Not this time . On the contrary , the deterioration of the relationship nowadays has developed a momentum of its own . There are four reasons for this . First , the \u201c loss \u201d of the Cold War , and with it imperial and superpower status , has created a deep and so far unresolved crisis in the collective mentality of Russia\u2019s political class . Russian leaders continue to perceive the West as a phantom enemy in opposition to which all the traditional mythologies of Russian foreign policy are being resurrected . Second , by the end of Vladimir Putin
 s second presidential term , Russia\u2019s modernizing dreams had been shattered . Modernization , indeed , simply turned out to be yet another redistribution of property to those on top , particularly those who came out of the St. Petersburg mayoral office and the Federal Security Bureau ( FSB ) . The image of the West as an enemy has become the only ideological excuse for Putin\u2019s model of the corporate state . Third , the soaring price of oil has made the Kremlin\u2019s inhabitants believe that they are all-powerful . Today\u2019s Russia , which thinks of itself as a \u201c great energy state , \u201d laughs at its previous meager desire to catch up with little Portugal in terms of living standards . Finally , a series of Western mistakes and misfortunes , a crisis in transatlantic relations , lack of leadership , and the growing threat of Islamic fundamentalism ( in both the Middle East and Europe ) have led Russian leaders to believe that the West is a sinking ship , to be abandoned as soon
  as possible . While this belief unfortunately does have some validity , it requires one very important caveat : Russia is part of that ship . Russia can make advances to Hamas , Hezbollah , and Iran , and it can remind the Arab world that the Soviet Union helped it develop and offered it protection in the United Nations Security Council . But in the eyes of Islamic extremists , Russia is part of the \u201c Satanic \u201d West \u2013 indeed , its most vulnerable part . Thus , it is Russia , with a soaring birth rate among its Muslim citizens that is the most attractive for expansion and take-over . But Russia\u2019s self-destructive confrontation with the West can be halted , and its centuries-old debate between Westernizers and the Slavophiles put to rest once and for all . This , however , will depend on Ukraine\u2019s success on the path of European development it chose in the Orange Revolution of 2004-2005 . Ukraine does , indeed , present a threat , but not to Russia\u2019s security , as Kremlin
  propagandists claim . The real threat is to the Putin model of a corporate , authoritarian state , unfriendly to the West . For the Kremlin\u2019s occupants , it is a matter of life and death that countries that were once part of the Soviet Union but chose a different model of development \u2013 Ukraine being the chief example \u2013 should never become attractive to ordinary Russians . The example posed by the Baltic nations does not threaten the Kremlin much , because they are perceived as foreign to the Russian psyche . Indeed , in Soviet films , Baltic actors were usually cast in the roles of Nazi generals and American spies . Ukrainians , on the other hand , are close to us in their culture and mentality . If they made a different choice , why can\u2019t we do the same ? Ukraine\u2019s success will mark the political death of Putinism , that squalid philosophy of \u201c KGB Capitalists . \u201d If Ukraine succeeds in its European choice , if it is able to make it work , it can settle the question 
 that has bedeviled Russian culture for centuries \u2013 Russia or the West ? So the best way to help Russia today is to support Ukraine\u2019s claim that it belongs to Europe and its institutions . This will influence Russia 's political mentality more than anything else . For if Russia\u2019s anti-Western paranoia continues and the Kremlin\u2019s Eurasian fantasy of allying with China lasts another 10-15 years , Russia will end up seeing China swallowing its Far East and Siberia . Indeed , the weakened Russia that will be Putin\u2019s legacy will then also lose the Northern Caucasus and the Volga region to their growing Muslim populations . The remaining Russian lands would then have no other choice but to attach themselves to Ukraine , which should by then have become a successful member of the European Union . After 1,000 years , Russia will have come full circle , returning to Kievan Rus after wandering on the roads of the Mongol hordes , empire , communism , and farcical Putinism . So Russia 
 now has a choice : Ukrainian plan A or Ukrainian plan B. 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/19syndicate_should-europe-regulate-sovereign-wealth-funds-.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/19syndicate_should-europe-regulate-sovereign-wealth-funds-.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/19syndicate_should-europe-regulate-sovereign-wealth-funds-.txt
new file mode 100644
index 0000000..8918954
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/19syndicate_should-europe-regulate-sovereign-wealth-funds-.txt
@@ -0,0 +1,2 @@
+
+Should Europe Regulate Sovereign Wealth Funds ? WIESBADEN \u2013 State-controlled investments from overseas \u2013 so-called sovereign wealth funds ( SWFs ) \u2013 are now the subject of intense debate . The United States and France have made their fears known . In Germany , too , the debate centers on SWFs \u2019 political and economic significance for the country\u2019s future . The problem has been exacerbated by the growing wealth of a number of countries , some of them formerly run by socialist or communist regimes . China , Russia , India , and the Gulf States have integrated their wealth into the global economy , to the immense benefit of world trade . The openness of Germany\u2019s markets makes them especially attractive to global trade . This openness will not change , yet there are some who now call for new safety fences \u2013 in other words , for protection . For example , Russian investors are interested in taking a massive share in the German-French aerospace company EADS , which is alrea
 dy 5%-owned by a Russian bank . For many , this proposal has underscored a change in investors \u2019 behavior . But what , exactly , has changed ? SWFs have been around for years . Among the first countries to invest their considerable state-owned funds were Kuwait , the United Arab Emirates , Norway , and Singapore . They invested , and still invest , their budgetary surpluses worldwide in government bonds and state-owned enterprises . Industrialized countries like the US and Japan also have so-called \u201c reserve funds . \u201d Some of these funds are huge . In the UAE , the Abu Dhabi Investment Authority has estimated capital assets of $875 billion , making it probably the world\u2019s largest state-owned investment company . In July 2007 , another rich UAE fund , Dubai International Capital , bought 3 % of EADS , after taking a stake of almost 2 % in the automotive manufacturer Daimler in January 2006. The Kuwait Investment Authority , also a state-owned fund , holds 7 % of Daimler . Sin
 gapore possesses two SWFs \u2013 Temasek-Holdings , with capital assets of roughly $100 billion , and the Government of Singapore Investment Corporation , with approximately $330 billion . Both funds are invested worldwide , including with the port operator PSA . Some funds are subject to considerable restrictions . Japan limits its state investments overseas to bonds , mostly those issued by the US . Until recently , China , which holds foreign currency reserves of more than $1.2 trillion ( the world\u2019s largest ) , followed this policy , too . But a $3 billion investment by the Chinese SWF in the US investment firm Blackstone suggests a more worryingly strategic investment policy , one that appears aimed at advancing its own industrial interests in certain markets . Russia , where the line between state-controlled and privately controlled companies is often blurry , has demonstrated this strategy in Europe . Indeed , Russian investments in aerospace , telecommunications , and , most 
 of all , in the energy sector , are the main source of concern for Germany . Are state-controlled investors now aiming primarily for strategic rather than purely financial returns ? Because SWFs \u2019 resources are so substantial , it is advisable to take precautions to avoid becoming a target of politically motivated market manipulation , or becoming economically and psychologically dependent on foreign governments \u2019 decisions . Most Western countries already have instruments to deter foreigners from making unwanted investments , not only in defense industries , but also in other sectors . But , while Germany\u2019s Foreign Trade and Payments Act protects against takeovers in the defense industry ( though the law needs strengthening ) elsewhere Germany has no system for examining investments by SWFs that may be strategically motivated . The International Monetary Fund now encourages more transparency by foreign investors , and has plans for a code of conduct . The EU Commission also f
 avors voluntary agreements aimed at strengthening transparency . Some SWFs now seem willing to engage in constructive dialogue . But assessing potential threats is not easy . Most investments are seen to benefit a country\u2019s economy , if not its security . We in Germany need to distinguish one from the other . Bills have been drafted that amount to amendments to the Foreign Trade and Payments Act and related regulations . While strengthening the act , they seek to avoid affecting the openness of the German economy . Under proposed new legislation , if a foreign investment in a German company amounts to more than 25 % , an assessment can be made of whether public order or safety might be threatened . In my view , this would address the concerns about SWFs , while not generally impeding investment because it would apply only in a very few cases . Germany has also drawn up a plan to protect its industries that is modeled on US regulation . Since 1988 , the US president can prohibit fo
 reign direct investment if it is seen as a threat to national security . An additional control was introduced last year , so now all direct investments in which a foreign government is involved are scrutinized by the Committee on Foreign Direct Investment . The principle of reciprocity should clearly apply to transnational investments . Germany is open to foreign investors , but in return we Germans demand the same market access abroad . Much remains to be done in this area even in Europe , as Germany\u2019s own experiences with France and Spain demonstrate . In China and almost all Middle Eastern countries , foreigners are restricted to minority shareholdings and must contend with high import duties and numerous non-tariff barriers . Protective measures must remain the exception rather than the rule . We Europeans must accept the challenges of global competition , and transnational investments are the basis of thriving economic development at home and abroad . Nevertheless , we must n
 ot allow ourselves to become the passive economic playthings of other nations , or of big state-owned enterprises . We must play an active part in shaping globalization , and that means crafting appropriate rules . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/1syndicate_china-and-a-new-balance-of-power.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/1syndicate_china-and-a-new-balance-of-power.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/1syndicate_china-and-a-new-balance-of-power.txt
new file mode 100644
index 0000000..8626fe4
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/1syndicate_china-and-a-new-balance-of-power.txt
@@ -0,0 +1,2 @@
+
+China and a New Balance of Power SHANGHAI \u2013 The last rival superpower to the United States , the Soviet Union , collapsed in 1991. But , apart from its military strength , the USSR was never really powerful enough to counterbalance US influence . In the late 1980\u2019s , Japan seemed capable of challenging America\u2019s industrial leadership , but by the 1990\u2019s it lost its competitive edge . China might itself wish to be a major force in a multi-polar world , but has been plagued by its lack of overall strength . Given these realities , China sees the expanding European Union as a likely counterweight to unchecked US power . In terms of economic output , today\u2019s EU is on par with the US . But it has yet to build a strong defense system that can respond effectively either to regional contingencies or to global needs . What would be the strength of such a system if and when it is developed , and how will it compare with that of the US ? In assessing America\u2019s strength , China foll
 ows US debates on the merits of a uni-polar or a multi-polar world with keen interest . Some Americans favor a uni-polar system in which the US dominates . Such a Pax Americana would cost the country less to sustain , but the world would worry if America adopts a wrong policy , as has been the case in Iraq . The US certainly has a right to curb terrorists like those who staged the attack on September 11 , 2001. But the \u201c war on terror \u201d did not warrant the decision to attack a sovereign state and topple its government on the flawed presumption that it housed weapons of mass destruction and was linked to the 2001 attacks . The US went to war despite strong opposition from France , Germany , Russia , China , and other United Nations Security Council members . France and Germany have since sought to restore good relations with the US , but continue to disapprove of the war . The Middle East , always a region of concern for Europe , has become even less stable as a result of the war
  . American leadership has been one of the war\u2019s casualties . In China , the concept of an independent European defense \u2013 embodied in the European Security and Defense Policy ( ESDP ) , as well as the EU\u2019s Rapid Reaction Force ( RRF ) and various police missions \u2013 is thus generally seen as necessary and effective in a multi-polar world system . China observes that on most security matters , the ESDP will ( for now ) mimic American security interests , because the EU and US share fundamental values of human rights and democracy . Moreover , many EU members have simultaneously pledged their defense capabilities to NATO , which has been led by the US since its creation . But the ESDP , once fully developed , will not necessarily follow America\u2019s lead unconditionally , especially if American policy continues to deviate from the norm of international law , as in Iraq . As a result of acquiring a pan-European defense capacity , the Europeans are likely to play a more independent
  role than at present in managing intra-European security relations and carrying out global missions . China welcomes this expanded security role for the EU . Although it remains wary of international intervention by the EU , the substance and pattern of ESDP operations are likely to win China\u2019s respect for several reasons . First , China\u2019s leaders note that the ESDP gives priority to the legitimacy of its missions . So far , all ESDP missions have respected international law and governmental arrangements among disputing parties . Most of its military or police missions have been based on UN Security Council resolutions , and ESDP missions outside Europe have been at the invitation of local authorities . Of course , ESDP security missions , even those in accordance with Security Council resolutions , may not operate within the UN system \u2013 the EU prefers its own independent leadership . The ESDP does not necessarily require Security Council authorization as a condition for its 
 missions , and it retains an independent role in executing them . However , when compared with the US , the ESDP ( in Chinese eyes ) cares far more about international legitimacy , backed by the authority of the UN . Second , the ESDP is concerned with good governance and institution building . The EU is keen to restore or establish human rights , stability , and prosperity . Outside Europe , the ESDP acts to enhance governance rather than promote regime change , and the EU has often assisted governments \u2019 efforts to improve security conditions . Third , the ESDP is open to international cooperation . The EU works with either non-EU states , such as NATO members like Canada , Norway , and Turkey , or applicants for EU membership . The ESDP cooperates with the UN and other regional organizations , such as the African Union and ASEAN . When the EU reaches out , it tends to play a leading role in these collaborations . There are good reasons to expect that China will continue to acce
 pt an independent European security mechanism . There is little concern , if any , about the ESDP intervening in internal Chinese affairs , such as Taiwan . Indeed , China wants a strong and independent Europe , and from that vantage point it is not too early for China to envisage a truly multi-polar global system . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/20syndicate_the--browning--of-african-technology.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/20syndicate_the--browning--of-african-technology.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/20syndicate_the--browning--of-african-technology.txt
new file mode 100644
index 0000000..483d11d
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/20syndicate_the--browning--of-african-technology.txt
@@ -0,0 +1,2 @@
+
+The \u201c Browning \u201d of African Technology Forget MIT . Hello , Tsing Hua University . For Clothilde Tingiri , a hot young programmer at Rwanda\u2019s top software company , dreams of Beijing , not Cambridge , animate her ambitions . Desperate for more education , this fall she plans to attend graduate school for computer science \u2013 in China , not America . The Chinese are no strangers to Rwanda . Near Tingiri\u2019s office , Rwanda\u2019s largest telecom company , Rwandatel , is installing new wireless telephony equipment made by Huawei of Shenzen . Africa boasts the world\u2019s fastest-growing market for wireless telephony , and Huawei \u2013 with offices in 14 African countries \u2013 is running away with the business , sending scores of engineers into the bush to bring a new generation of low-cost technology to some of the planet\u2019s poorest people . Motivated by profit and market share rather than philanthropy , Huawei is outpacing American and European rivals through lower prices , faster act
 ion , and a greater willingness to work in difficult environments . According to Chris Lundh , the American chief of Rwandatel , \u201c That\u2019s the way things work in Africa now . The Chinese do it all . \u201d Well , not quite . Across sub-Saharan Africa , engineers from India \u2013 armed with appropriate technologies honed in their home market \u2013 are also making their mark . India supplies Africa with computer-education courses , the most reliable water pumps , low-cost rice-milling equipment , and dozens of other technologies . The sudden influx of Chinese and Indian technologies represents the \u201c browning \u201d of African technology , which has long been the domain of \u201c white \u201d Americans and Europeans who want to apply their saving hand to African problems . \u201c It is a tectonic shift to the East with shattering implications , \u201d says Calestous Juma , a Kenyan professor at Harvard University who advises the African Union on technology policy . One big change is in education . Ther
 e are roughly 2,000 African students in China , most of whom are pursuing engineering and science courses . According to Juma , that number is expected to double over the next two years , making China \u201c Africa\u2019s leading destination for science and engineering education . \u201d The \u201c browning \u201d of technology in Africa is only in its infancy , but the shift is likely to accelerate . Chinese and Indian engineers hail from places that have much more in common with nitty-gritty Africa than comfortable Silicon Valley or Cambridge . Africa also offers a testing ground for Asian-designed technologies that are not yet ready for US or European markets . A good example is a solar-powered cooking stove from India , which has experimented with such stoves for decades . Wood-burning stoves are responsible for much of Africa\u2019s deforestation , and , in many African cities , where wood accounts for the majority of cooking fuel , its price is soaring . The Indian stove is clearly a work-in-pr
 ogress ; it is too bulky and not durable enough to survive the rigors of an African village . But with India\u2019s vast internal market , many designers have an incentive to improve it . How many designers in America or Europe can say the same ? Of course , technology transfer from China and India could be a mere smokescreen for a new \u201c brown imperialism \u201d aimed at exploiting African oil , food , and minerals . In recent years , China\u2019s government alone has invested billions of dollars in African infrastructure and resource extraction , raising suspicions that a new scramble for Africa is underway . But Africans genuinely need foreign technology , and the Chinese , in particular , are pushing hard \u2013 even flamboyantly \u2013 to fill the gap . This year , Nigeria\u2019s government bought a Chinese-made satellite , and even paid the Chinese to launch it into space in May . China was so eager to provide space technology to Africa\u2019s most populous country that it beat out 21 other bidde
 rs for a contract worth $300 million . China\u2019s technology inroads are usually less dramatic , but no less telling . In African medicine , Chinese herbs and pharmaceuticals are quietly gaining share . For example , the Chinese-made anti-malarial drug artesunate has become part of the standard treatment within just a few years . Likewise , Chinese mastery over ultra-small , cheap \u201c micro-hydro \u201d dams , which can generate tiny amounts of electricity from mere trickles of water , appeals to power-short , river-rich Africans . Tens of thousands of micro-hydro systems operate in China , and nearly none in Africa . Americans do-gooders like Nicholas Negroponte , with his $100 laptop , have identified the right problem : Africa is way behind technologically and rapid leap-frogging is possible . But Chinese and Indian scientists argue that Africa can benefit from a changing of the technological guard . They may be right . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/21syndicate_the-china-question.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/21syndicate_the-china-question.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/21syndicate_the-china-question.txt
new file mode 100644
index 0000000..c725a0c
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/synd/21syndicate_the-china-question.txt
@@ -0,0 +1,2 @@
+
+The China Question President Bush\u2019s recent visit to Asia made little news \u2013 by design . But that\u2019s because Bush didn\u2019t begin to address the issue that is looming ever larger in the region : the changing face of security in Asia in view of China\u2019s growing economic and military might . This summer , for example , China and Russia conducted their first-ever grand-scale joint military exercise . This was followed by Russian news reports that China , Russia , and India would conduct trilateral military exercises , named \u201c Indira 2005 , \u201d on the same scale before the end of this year . In the past , such a combination of countries was almost unthinkable , and these exercises cannot be explained away as simple \u201c one-off \u201d affairs with little resonance . Instead , they reflect China\u2019s long-term strategic goal of establishing hegemony across Asia . One tool of this ambition is the Shanghai Cooperation Organization ( SCO ) , under which the Sino-Russian exercises took plac
 e . Established in June 2001 , the SCO includes China , Russia , Kazakhstan , Kyrgyzstan , Tajikistan , and Uzbekistan . The SCO\u2019s original purpose was to mitigate tensions on the borders of China and the Central Asian countries after the collapse of the Soviet Union and the arrival of the United States military with the war in Afghanistan . China regards the SCO as a stage for broadening its influence over a vast region , ranging from the Asia-Pacific to Southwest Asia , the Middle East , East Africa , and the Indian Ocean . Indeed , its members include about 45 % of the world\u2019s population , and 28 % of the landmass ranging across the Eurasian continent . China\u2019s active leadership of the SCO has resulted in policies that it favors . Gradually , the SCO shifted its focus to fighting Islamic radicals . Nowadays , however , the SCO is often used as a forum to campaign against supposed American unilateralism and to provide a united front \u2013 especially between China and Russia 
  against the US with respect to security and arms-reduction issues in the region . This includes joint anti-terror training and demands to reduce US forces in the region , particularly from Uzbekistan and Kyrgyzstan . The SCO provides China not only with a platform to confront the existing US-led alliance in the Asia-Pacific region , but is increasingly being used to prevent the formation of a US-led network to restrain China\u2019s advance . Ultimately , it is feared that the SCO could develop into a military alliance similar to the Warsaw Pact of the Cold War era , with an embryonic \u201c Great China Union \u201d at its core . But China\u2019s regional diplomacy goes far beyond the SCO . It seizes every opportunity that comes its way , including the Six Party Talks on North Korea\u2019s nuclear ambitions , to emphasize its centrality to the settlement of any and all Asian issues . Moreover , it continues to build its \u201c string of pearls \u201d of military bases at every key point on maritime tra
 nsportation routes along the \u201c arc of instability \u201d from the Middle East to China\u2019s coast . No one seems to know how to respond to China\u2019s diplomatic and military muscle flexing in Asia , for the extent of China\u2019s ambitions remains utterly unclear . But , while everyone else ponders China\u2019s motives , its government is acting . Indeed , the UK\u2019s premier security think tank , the Institute for International Strategic Studies , recently warned that , while the world focuses on the fight against international terrorism and the unfolding events in the Middle East , China is rapidly expanding its influence from Asia to Africa . The \u201c pearls \u201d in Africa include Sudan , Angola , Algeria , Gabon , Namibia , Zambia , Tanzania , Zimbabwe , Uganda , Djibouti , Mali , Central Africa , Liberia , Ethiopia , Mozambique , Sierra Leone , and the Democratic Republic of Congo . In each country , China is nurturing special military and commercial relations intended to promote loyalty t
 o Chinese interests . As in Asia , there is a pattern at work : growing Chinese influence begets increased support for Chinese policies . Of course , it\u2019s a two-way street . Whenever complaints come up in the United Nations \u2019 Human Rights Committee , China can count on the support of many African countries that have their own human rights problems . Even the selection of Beijing as the site of the 2008 Olympics benefited from \u201c African votes . \u201d And China has publicly stated that it will back African nations in potential disputes at the World Trade Organization ( WTO ) and other international organizations . Similarly , many African states now seem to be leaning heavily toward China in its dispute with Taiwan . When Japan\u2019s government tried to become a permanent member of the UN Security Council , few African countries backed its bid , despite receiving economic aid for decades . China likes to boast of its \u201c peaceful rise . \u201d But the rise of Bismarck\u2019s Germany at th
 e end of the nineteenth century was also peaceful \u2013 for a while . The question is not whether China rises to great-power status peacefully , but whether it intends to remain peaceful when it gets there . Just as the world confronted the \u201c German Question \u201d 125 years ago , it is now confronting the \u201c China Question . \u201d We need a better answer this time . 
\ No newline at end of file


[06/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/A/179un_A_C3_61_L17.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/un/A/179un_A_C3_61_L17.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/A/179un_A_C3_61_L17.txt
new file mode 100644
index 0000000..d1e78ae
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/A/179un_A_C3_61_L17.txt
@@ -0,0 +1,2 @@
+
+Sixty-first session Third Committee Agenda item 68 Report of the Human Rights Council Andorra , Argentina , Austria , Bangladesh , Belgium , Benin , Bolivia , Brazil , Bulgaria , Burundi , Cameroon , Cape Verde , Chile , Congo , Croatia , Cuba , Cyprus , Czech Republic , Ecuador , Estonia , Finland , France , Gabon , Georgia , Germany , Greece , Guinea-Bissau , Haiti , Honduras , Hungary , Iceland , Ireland , Italy , Latvia , Liechtenstein , Lithuania , Luxembourg , Madagascar , Malawi , Mali , Malta , Mauritius , Mexico , Monaco , Morocco , Netherlands , Nigeria , Norway , Panama , Paraguay , Peru , Poland , Portugal , Republic of Korea , Romania , Slovakia , Slovenia , South Africa , Spain , Switzerland , Uganda , United Republic of Tanzania and Venezuela ( Bolivarian Republic of ) : draft resolution* International Convention for the Protection of All Persons from Enforced Disappearance The General Assembly , Taking note of Human Rights Council resolution 1/1 of 29 June 2006 , by 
 which the Council adopted the International Convention for the Protection of All Persons from Enforced Disappearance , Hails the Council 's adoption of the International Convention for the Protection of All Persons from Enforced Disappearance ; 1 Adopts and opens for signature , ratification and accession the International Convention for the Protection of All Persons from Enforced Disappearance , the text of which is annexed to the present resolution ; Recommends that the Convention be opened for signature at a signing ceremony in Paris . Annex International Convention for the Protection of All Persons from Enforced Disappearance Preamble The States Parties to this Convention , Considering the obligation of States under the Charter of the United Nations to promote universal respect for , and observance of , human rights and fundamental freedoms , Having regard to the Universal Declaration of Human Rights , Recalling the International Covenant on Economic , Social and Cultural Rights
  , the International Covenant on Civil and Political Rights and the other relevant international instruments in the fields of human rights , humanitarian law and international criminal law , Also recalling the Declaration on the Protection of All Persons from Enforced Disappearance adopted by the General Assembly of the United Nations in its resolution 47/133 of 18 December 1992 , Aware of the extreme seriousness of enforced disappearance , which constitutes a crime and , in certain circumstances defined in international law , a crime against humanity , Determined to prevent enforced disappearances and to combat impunity for the crime of enforced disappearance , Considering the right of any person not to be subjected to enforced disappearance , the right of victims to justice and to reparation , Affirming the right of any victim to know the truth about the circumstances of an enforced disappearance and the fate of the disappeared person , and the right to freedom to seek , receive a
 nd impart information to this end , Have agreed on the following articles : Part I Article 1 No one shall be subjected to enforced disappearance . No exceptional circumstances whatsoever , whether a state of war or a threat of war , internal political instability or any other public emergency , may be invoked as a justification for enforced disappearance . Article 2 For the purposes of this Convention , \u201c enforced disappearance \u201d is considered to be the arrest , detention , abduction or any other form of deprivation of liberty by agents of the State or by persons or groups of persons acting with the authorization , support or acquiescence of the State , followed by a refusal to acknowledge the deprivation of liberty or by concealment of the fate or whereabouts of the disappeared person , which places such a person outside the protection of the law . Article 3 Each State Party shall take appropriate measures to investigate acts defined in article 2 committed by persons or groups 
 of persons acting without the authorization , support or acquiescence of the State and to bring those responsible to justice . Article 4 Each State Party shall take the necessary measures to ensure that enforced disappearance constitutes an offence under its criminal law . Article 5 The widespread or systematic practice of enforced disappearance constitutes a crime against humanity as defined in applicable international law and shall attract the consequences provided for under such applicable international law . Article 6 Each State Party shall take the necessary measures to hold criminally responsible at least : ( a ) Any person who commits , orders , solicits or induces the commission of , attempts to commit , is an accomplice to or participates in an enforced disappearance ; ( b ) A superior who : ( i ) Knew , or consciously disregarded information which clearly indicated , that subordinates under his or her effective authority and control were committing or about to commit a cri
 me of enforced disappearance ; ( ii ) Exercised effective responsibility for and control over activities which were concerned with the crime of enforced disappearance ; and ( iii ) Failed to take all necessary and reasonable measures within his or her power to prevent or repress the commission of an enforced disappearance or to submit the matter to the competent authorities for investigation and prosecution ; ( c ) Subparagraph ( b ) above is without prejudice to the higher standards of responsibility applicable under relevant international law to a military commander or to a person effectively acting as a military commander . No order or instruction from any public authority , civilian , military or other , may be invoked to justify an offence of enforced disappearance . Article 7 Each State Party shall make the offence of enforced disappearance punishable by appropriate penalties which take into account its extreme seriousness . Each State Party may establish : ( a ) Mitigating ci
 rcumstances , in particular for persons who , having been implicated in the commission of an enforced disappearance , effectively contribute to bringing the disappeared person forward alive or make it possible to clarify cases of enforced disappearance or to identify the perpetrators of an enforced disappearance ; ( b ) Without prejudice to other criminal procedures , aggravating circumstances , in particular in the event of the death of the disappeared person or the commission of an enforced disappearance in respect of pregnant women , minors , persons with disabilities or other particularly vulnerable persons . Article 8 Without prejudice to article 5 , A State Party which applies a statute of limitations in respect of enforced disappearance shall take the necessary measures to ensure that the term of limitation for criminal proceedings : ( a ) Is of long duration and is proportionate to the extreme seriousness of this offence ; ( b ) Commences from the moment when the offence of 
 enforced disappearance ceases , taking into account its continuous nature . Each State Party shall guarantee the right of victims of enforced disappearance to an effective remedy during the term of limitation . Article 9 Each State Party shall take the necessary measures to establish its competence to exercise jurisdiction over the offence of enforced disappearance : ( a ) When the offence is committed in any territory under its jurisdiction or on board a ship or aircraft registered in that State ; ( b ) When the alleged offender is one of its nationals ; ( c ) When the disappeared person is one of its nationals and the State Party considers it appropriate . Each State Party shall likewise take such measures as may be necessary to establish its competence to exercise jurisdiction over the offence of enforced disappearance when the alleged offender is present in any territory under its jurisdiction , unless it extradites or surrenders him or her to another State in accordance with it
 s international obligations or surrenders him or her to an international criminal tribunal whose jurisdiction it has recognized . This Convention does not exclude any additional criminal jurisdiction exercised in accordance with national law . Article 10 Upon being satisfied , after an examination of the information available to it , that the circumstances so warrant , any State Party in whose territory a person suspected of having committed an offence of enforced disappearance is present shall take him or her into custody or take such other legal measures as are necessary to ensure his or her presence . The custody and other legal measures shall be as provided for in the law of that State Party but may be maintained only for such time as is necessary to ensure the person 's presence at criminal , surrender or extradition proceedings . A State Party which has taken the measures referred to in paragraph 1 of this article shall immediately carry out a preliminary inquiry or investigat
 ions to establish the facts . It shall notify the States Parties referred to in article 9 , paragraph 1 , of the measures it has taken in pursuance of paragraph 1 of this article , including detention and the circumstances warranting detention , and of the findings of its preliminary inquiry or its investigations , indicating whether it intends to exercise its jurisdiction . Any person in custody pursuant to paragraph 1 of this article may communicate immediately with the nearest appropriate representative of the State of which he or she is a national , or , if he or she is a stateless person , with the representative of the State where he or she usually resides . Article 11 The State Party in the territory under whose jurisdiction a person alleged to have committed an offence of enforced disappearance is found shall , if it does not extradite that person or surrender him or her to another State in accordance with its international obligations or surrender him or her to an internati
 onal criminal tribunal whose jurisdiction it has recognized , submit the case to its competent authorities for the purpose of prosecution . These authorities shall take their decision in the same manner as in the case of any ordinary offence of a serious nature under the law of that State Party . In the cases referred to in article 9 , paragraph 2 , the standards of evidence required for prosecution and conviction shall in no way be less stringent than those which apply in the cases referred to in article 9 , paragraph 1. Any person against whom proceedings are brought in connection with an offence of enforced disappearance shall be guaranteed fair treatment at all stages of the proceedings . Any person tried for an offence of enforced disappearance shall benefit from a fair trial before a competent , independent and impartial court or tribunal established by law . Article 12 Each State Party shall ensure that any individual who alleges that a person has been subjected to enforced d
 isappearance has the right to report the facts to the competent authorities , which shall examine the allegation promptly and impartially and , where necessary , undertake without delay a thorough and impartial investigation . Appropriate steps shall be taken , where necessary , to ensure that the complainant , witnesses , relatives of the disappeared person and their defence counsel , as well as persons participating in the investigation , are protected against all ill-treatment or intimidation as a consequence of the complaint or any evidence given . Where there are reasonable grounds for believing that a person has been subjected to enforced disappearance , the authorities referred to in paragraph 1 of this article shall undertake an investigation , even if there has been no formal complaint . Each State Party shall ensure that the authorities referred to in paragraph 1 of this article : ( a ) Have the necessary powers and resources to conduct the investigation effectively , incl
 uding access to the documentation and other information relevant to their investigation ; ( b ) Have access , if necessary with the prior authorization of a judicial authority , which shall rule promptly on the matter , to any place of detention or any other place where there are reasonable grounds to believe that the disappeared person may be present . Each State Party shall take the necessary measures to prevent and sanction acts that hinder the conduct of an investigation . It shall ensure in particular that persons suspected of having committed an offence of enforced disappearance are not in a position to influence the progress of an investigation by means of pressure or acts of intimidation or reprisal aimed at the complainant , witnesses , relatives of the disappeared person or their defence counsel , or at persons participating in the investigation . Article 13 For the purposes of extradition between States Parties , the offence of enforced disappearance shall not be regarded
  as a political offence or as an offence connected with a political offence or as an offence inspired by political motives . Accordingly , a request for extradition based on such an offence may not be refused on these grounds alone . The offence of enforced disappearance shall be deemed to be included as an extraditable offence in any extradition treaty existing between States Parties before the entry into force of this Convention . States Parties undertake to include the offence of enforced disappearance as an extraditable offence in any extradition treaty subsequently to be concluded between them . If a State Party which makes extradition conditional on the existence of a treaty receives a request for extradition from another State Party with which it has no extradition treaty , it may consider this Convention as the necessary legal basis for extradition in respect of the offence of enforced disappearance . States Parties which do not make extradition conditional on the existence 
 of a treaty shall recognize the offence of enforced disappearance as an extraditable offence between themselves . Extradition shall , in all cases , be subject to the conditions provided for by the law of the requested State Party or by applicable extradition treaties , including , in particular , conditions relating to the minimum penalty requirement for extradition and the grounds upon which the requested State Party may refuse extradition or make it subject to certain conditions . Nothing in this Convention shall be interpreted as imposing an obligation to extradite if the requested State Party has substantial grounds for believing that the request has been made for the purpose of prosecuting or punishing a person on account of that person 's sex , race , religion , nationality , ethnic origin , political opinions or membership of a particular social group , or that compliance with the request would cause harm to that person for any one of these reasons . Article 14 States Partie
 s shall afford one another the greatest measure of mutual legal assistance in connection with criminal proceedings brought in respect of an offence of enforced disappearance , including the supply of all evidence at their disposal that is necessary for the proceedings . Such mutual legal assistance shall be subject to the conditions provided for by the domestic law of the requested State Party or by applicable treaties on mutual legal assistance , including , in particular , the conditions in relation to the grounds upon which the requested State Party may refuse to grant mutual legal assistance or may make it subject to conditions . Article 15 States Parties shall cooperate with each other and shall afford one another the greatest measure of mutual assistance with a view to assisting victims of enforced disappearance , and in searching for , locating and releasing disappeared persons and , in the event of death , in exhuming and identifying them and returning their remains . Articl
 e 16 No State Party shall expel , return ( \u201c refouler \u201d ) , surrender or extradite a person to another State where there are substantial grounds for believing that he or she would be in danger of being subjected to enforced disappearance . For the purpose of determining whether there are such grounds , the competent authorities shall take into account all relevant considerations , including , where applicable , the existence in the State concerned of a consistent pattern of gross , flagrant or mass violations of human rights or of serious violations of international humanitarian law . Article 17 No one shall be held in secret detention . Without prejudice to other international obligations of the State Party with regard to the deprivation of liberty , each State Party shall , in its legislation : ( a ) Establish the conditions under which orders of deprivation of liberty may be given ; ( b ) Indicate those authorities authorized to order the deprivation of liberty ; ( c ) Guaran
 tee that any person deprived of liberty shall be held solely in officially recognized and supervised places of deprivation of liberty ; ( d ) Guarantee that any person deprived of liberty shall be authorized to communicate with and be visited by his or her family , counsel or any other person of his or her choice , subject only to the conditions established by law , or , if he or she is a foreigner , to communicate with his or her consular authorities , in accordance with applicable international law ; ( e ) Guarantee access by the competent and legally authorized authorities and institutions to the places where persons are deprived of liberty , if necessary with prior authorization from a judicial authority ; ( f ) Guarantee that any person deprived of liberty or , in the case of a suspected enforced disappearance , since the person deprived of liberty is not able to exercise this right , any persons with a legitimate interest , such as relatives of the person deprived of liberty ,
  their representatives or their counsel , shall , in all circumstances , be entitled to take proceedings before a court , in order that the court may decide without delay on the lawfulness of the deprivation of liberty and order the person 's release if such deprivation of liberty is not lawful . Each State Party shall assure the compilation and maintenance of one or more up-to-date official registers and/or records of persons deprived of liberty , which shall be made promptly available , upon request , to any judicial or other competent authority or institution authorized for that purpose by the law of the State Party concerned or any relevant international legal instrument to which the State concerned is a party . The information contained therein shall include , as a minimum : ( a ) The identity of the person deprived of liberty ; ( b ) The date , time and place where the person was deprived of liberty and the identity of the authority that deprived the person of liberty ; ( c ) 
 The authority that ordered the deprivation of liberty and the grounds for the deprivation of liberty ; ( d ) The authority responsible for supervising the deprivation of liberty ; ( e ) The place of deprivation of liberty , the date and time of admission to the place of deprivation of liberty and the authority responsible for the place of deprivation of liberty ; ( f ) Elements relating to the state of health of the person deprived of liberty ; ( g ) In the event of death during the deprivation of liberty , the circumstances and cause of death and the destination of the remains ; ( h ) The date and time of release or transfer to another place of detention , the destination and the authority responsible for the transfer . Article 18 Subject to articles 19 and 20 , each State Party shall guarantee to any person with a legitimate interest in this information , such as relatives of the person deprived of liberty , their representatives or their counsel , access to at least the following
  information : ( a ) The authority that ordered the deprivation of liberty ; ( b ) The date , time and place where the person was deprived of liberty and admitted to the place of deprivation of liberty ; ( c ) The authority responsible for supervising the deprivation of liberty ; ( d ) The whereabouts of the person deprived of liberty , including , in the event of a transfer to another place of deprivation of liberty , the destination and the authority responsible for the transfer ; ( e ) The date , time and place of release ; ( f ) Elements relating to the state of health of the person deprived of liberty ; ( g ) In the event of death during the deprivation of liberty , the circumstances and cause of death and the destination of the remains . Appropriate measures shall be taken , where necessary , to protect the persons referred to in paragraph 1 of this article , as well as persons participating in the investigation , from any ill-treatment , intimidation or sanction as a result o
 f the search for information concerning a person deprived of liberty . Article 19 Personal information , including medical and genetic data , which is collected and/or transmitted within the framework of the search for a disappeared person shall not be used or made available for purposes other than the search for the disappeared person . This is without prejudice to the use of such information in criminal proceedings relating to an offence of enforced disappearance or the exercise of the right to obtain reparation . The collection , processing , use and storage of personal information , including medical and genetic data , shall not infringe or have the effect of infringing the human rights , fundamental freedoms or human dignity of an individual . Article 20 Only where a person is under the protection of the law and the deprivation of liberty is subject to judicial control may the right to information referred to in article 18 be restricted , on an exceptional basis , where strictl
 y necessary and where provided for by law , and if the transmission of the information would adversely affect the privacy or safety of the person , hinder a criminal investigation , or for other equivalent reasons in accordance with the law , and in conformity with applicable international law and with the objectives of this Convention . In no case shall there be restrictions on the right to information referred to in article 18 that could constitute conduct defined in article 2 or be in violation of article 17 , paragraph 1. Without prejudice to consideration of the lawfulness of the deprivation of a person 's liberty , States Parties shall guarantee to the persons referred to in article 18 , paragraph 1 , the right to a prompt and effective judicial remedy as a means of obtaining without delay the information referred to in article 18 , paragraph 1. This right to a remedy may not be suspended or restricted in any circumstances . Article 21 Each State Party shall take the necessary
  measures to ensure that persons deprived of liberty are released in a manner permitting reliable verification that they have actually been released . Each State Party shall also take the necessary measures to assure the physical integrity of such persons and their ability to exercise fully their rights at the time of release , without prejudice to any obligations to which such persons may be subject under national law . Article 22 Without prejudice to article 6 , each State Party shall take the necessary measures to prevent and impose sanctions for the following conduct : ( a ) Delaying or obstructing the remedies referred to in article 17 , paragraph 2 ( f ) , and article 20 , paragraph 2 ; ( b ) Failure to record the deprivation of liberty of any person , or the recording of any information which the official responsible for the official register knew or should have known to be inaccurate ; ( c ) Refusal to provide information on the deprivation of liberty of a person , or the pr
 ovision of inaccurate information , even though the legal requirements for providing such information have been met . Article 23 Each State Party shall ensure that the training of law enforcement personnel , civil or military , medical personnel , public officials and other persons who may be involved in the custody or treatment of any person deprived of liberty includes the necessary education and information regarding the relevant provisions of this Convention , in order to : ( a ) Prevent the involvement of such officials in enforced disappearances ; ( b ) Emphasize the importance of prevention and investigations in relation to enforced disappearances ; ( c ) Ensure that the urgent need to resolve cases of enforced disappearance is recognized . Each State Party shall ensure that orders or instructions prescribing , authorizing or encouraging enforced disappearance are prohibited . Each State Party shall guarantee that a person who refuses to obey such an order will not be punishe
 d . Each State Party shall take the necessary measures to ensure that the persons referred to in paragraph 1 of this article who have reason to believe that an enforced disappearance has occurred or is planned report the matter to their superiors and , where necessary , to the appropriate authorities or bodies vested with powers of review or remedy . Article 24 For the purposes of this Convention , \u201c victim \u201d means the disappeared person and any individual who has suffered harm as the direct result of an enforced disappearance . Each victim has the right to know the truth regarding the circumstances of the enforced disappearance , the progress and results of the investigation and the fate of the disappeared person . Each State Party shall take appropriate measures in this regard . Each State Party shall take all appropriate measures to search for , locate and release disappeared persons and , in the event of death , to locate , respect and return their remains . Each State Party
  shall ensure in its legal system that the victims of enforced disappearance have the right to obtain reparation and prompt , fair and adequate compensation . The right to obtain reparation referred to in paragraph 4 of this article covers material and moral damages and , where appropriate , other forms of reparation such as : ( a ) Restitution ; ( b ) Rehabilitation ; ( c ) Satisfaction , including restoration of dignity and reputation ; ( d ) Guarantees of non-repetition . Without prejudice to the obligation to continue the investigation until the fate of the disappeared person has been clarified , each State Party shall take the appropriate steps with regard to the legal situation of disappeared persons whose fate has not been clarified and that of their relatives , in fields such as social welfare , financial matters , family law and property rights . Each State Party shall guarantee the right to form and participate freely in organizations and associations concerned with attemp
 ting to establish the circumstances of enforced disappearances and the fate of disappeared persons , and to assist victims of enforced disappearance . Article 25 Each State Party shall take the necessary measures to prevent and punish under its criminal law : ( a ) The wrongful removal of children who are subjected to enforced disappearance , children whose father , mother or legal guardian is subjected to enforced disappearance or children born during the captivity of a mother subjected to enforced disappearance ; ( b ) The falsification , concealment or destruction of documents attesting to the true identity of the children referred to in subparagraph ( a ) above . Each State Party shall take the necessary measures to search for and identify the children referred to in paragraph 1 ( a ) of this article and to return them to their families of origin , in accordance with legal procedures and applicable international agreements . States Parties shall assist one another in searching f
 or , identifying and locating the children referred to in paragraph 1 ( a ) of this article . Given the need to protect the best interests of the children referred to in paragraph 1 ( a ) of this article and their right to preserve , or to have re-established , their identity , including their nationality , name and family relations as recognized by law , States Parties which recognize a system of adoption or other form of placement of children shall have legal procedures in place to review the adoption or placement procedure , and , where appropriate , to annul any adoption or placement of children that originated in an enforced disappearance . In all cases , and in particular in all matters relating to this article , the best interests of the child shall be a primary consideration , and a child who is capable of forming his or her own views shall have the right to express those views freely , the views of the child being given due weight in accordance with the age and maturity of 
 the child . Part II Article 26 A Committee on Enforced Disappearances ( hereinafter referred to as \u201c the Committee \u201d ) shall be established to carry out the functions provided for under this Convention . The Committee shall consist of ten experts of high moral character and recognized competence in the field of human rights , who shall serve in their personal capacity and be independent and impartial . The members of the Committee shall be elected by the States Parties according to equitable geographical distribution . Due account shall be taken of the usefulness of the participation in the work of the Committee of persons having relevant legal experience and of balanced gender representation . The members of the Committee shall be elected by secret ballot from a list of persons nominated by States Parties from among their nationals , at biennial meetings of the States Parties convened by the Secretary General of the United Nations for this purpose . At those meetings , for whic
 h two thirds of the States Parties shall constitute a quorum , the persons elected to the Committee shall be those who obtain the largest number of votes and an absolute majority of the votes of the representatives of States Parties present and voting . The initial election shall be held no later than six months after the date of entry into force of this Convention . Four months before the date of each election , the Secretary-General of the United Nations shall address a letter to the States Parties inviting them to submit nominations within three months . The Secretary-General shall prepare a list in alphabetical order of all persons thus nominated , indicating the State Party which nominated each candidate , and shall submit this list to all States Parties . The members of the Committee shall be elected for a term of four years . They shall be eligible for re-election once . However , the term of five of the members elected at the first election shall expire at the end of two yea
 rs ; immediately after the first election , the names of these five members shall be chosen by lot by the chairman of the meeting referred to in paragraph 2 of this article . If a member of the Committee dies or resigns or for any other reason can no longer perform his or her Committee duties , the State Party which nominated him or her shall , in accordance with the criteria set out in paragraph 1 of this article , appoint another candidate from among its nationals to serve out his or her term , subject to the approval of the majority of the States Parties . Such approval shall be considered to have been obtained unless half or more of the States Parties respond negatively within six weeks of having been informed by the Secretary-General of the United Nations of the proposed appointment . The Committee shall establish its own rules of procedure . The Secretary-General of the United Nations shall provide the Committee with the necessary means , staff and facilities for the effective
  performance of its functions . The Secretary-General of the United Nations shall convene the initial meeting of the Committee . The members of the Committee shall be entitled to the facilities , privileges and immunities of experts on mission for the United Nations , as laid down in the relevant sections of the Convention on the Privileges and Immunities of the United Nations . Each State Party shall cooperate with the Committee and assist its members in the fulfilment of their mandate , to the extent of the Committee 's functions that the State Party has accepted . Article 27 A Conference of the States Parties will take place at the earliest four years and at the latest six years following the entry into force of this Convention to evaluate the functioning of the Committee and to decide , in accordance with the procedure described in article 44 , paragraph 2 , whether it is appropriate to transfer to another body \u2014 without excluding any possibility \u2014 the monitoring of this Con
 vention , in accordance with the functions defined in articles 28 to 36. Article 28 In the framework of the competencies granted by this Convention , the Committee shall cooperate with all relevant organs , offices and specialized agencies and funds of the United Nations , with the treaty bodies instituted by international instruments , with the special procedures of the United Nations and with the relevant regional intergovernmental organizations or bodies , as well as with all relevant State institutions , agencies or offices working towards the protection of all persons against enforced disappearances . As it discharges its mandate , the Committee shall consult other treaty bodies instituted by relevant international human rights instruments , in particular the Human Rights Committee instituted by the International Covenant on Civil and Political Rights , with a view to ensuring the consistency of their respective observations and recommendations . Article 29 Each State Party sha
 ll submit to the Committee , through the Secretary-General of the United Nations , a report on the measures taken to give effect to its obligations under this Convention , within two years after the entry into force of this Convention for the State Party concerned . The Secretary-General of the United Nations shall make this report available to all States Parties . Each report shall be considered by the Committee , which shall issue such comments , observations or recommendations as it may deem appropriate . The comments , observations or recommendations shall be communicated to the State Party concerned , which may respond to them , on its own initiative or at the request of the Committee . The Committee may also request States Parties to provide additional information on the implementation of this Convention . Article 30 A request that a disappeared person should be sought and found may be submitted to the Committee , as a matter of urgency , by relatives of the disappeared person
  or their legal representatives , their counsel or any person authorized by them , as well as by any other person having a legitimate interest . If the Committee considers that a request for urgent action submitted in pursuance of paragraph 1 of this article : ( a ) Is not manifestly unfounded ; ( b ) Does not constitute an abuse of the right of submission of such requests ; ( c ) Has already been duly presented to the competent bodies of the State Party concerned , such as those authorized to undertake investigations , where such a possibility exists ; ( d ) Is not incompatible with the provisions of this Convention ; and ( e ) The same matter is not being examined under another procedure of international investigation or settlement of the same nature ; it shall request the State Party concerned to provide it with information on the situation of the persons sought , within a time limit set by the Committee . In the light of the information provided by the State Party concerned in a
 ccordance with paragraph 2 of this article , the Committee may transmit recommendations to the State Party , including a request that the State Party should take all the necessary measures , including interim measures , to locate and protect the person concerned in accordance with this Convention and to inform the Committee , within a specified period of time , of measures taken , taking into account the urgency of the situation . The Committee shall inform the person submitting the urgent action request of its recommendations and of the information provided to it by the State as it becomes available . The Committee shall continue its efforts to work with the State Party concerned for as long as the fate of the person sought remains unresolved . The person presenting the request shall be kept informed . Article 31 A State Party may at the time of ratification of this Convention or at any time afterwards declare that it recognizes the competence of the Committee to receive and consid
 er communications from or on behalf of individuals subject to its jurisdiction claiming to be victims of a violation by this State Party of provisions of this Convention . The Committee shall not admit any communication concerning a State Party which has not made such a declaration . The Committee shall consider a communication inadmissible where : ( a ) The communication is anonymous ; ( b ) The communication constitutes an abuse of the right of submission of such communications or is incompatible with the provisions of this Convention ; ( c ) The same matter is being examined under another procedure of international investigation or settlement of the same nature ; or where ( d ) All effective available domestic remedies have not been exhausted . This rule shall not apply where the application of the remedies is unreasonably prolonged . If the Committee considers that the communication meets the requirements set out in paragraph 2 of this article , it shall transmit the communicati
 on to the State Party concerned , requesting it to provide observations and comments within a time limit set by the Committee . At any time after the receipt of a communication and before a determination on the merits has been reached , the Committee may transmit to the State Party concerned for its urgent consideration a request that the State Party will take such interim measures as may be necessary to avoid possible irreparable damage to the victims of the alleged violation . Where the Committee exercises its discretion , this does not imply a determination on admissibility or on the merits of the communication . The Committee shall hold closed meetings when examining communications under the present article . It shall inform the author of a communication of the responses provided by the State Party concerned . When the Committee decides to finalize the procedure , it shall communicate its views to the State Party and to the author of the communication . Article 32 A State Party 
 to this Convention may at any time declare that it recognizes the competence of the Committee to receive and consider communications in which a State Party claims that another State Party is not fulfilling its obligations under this Convention . The Committee shall not receive communications concerning a State Party which has not made such a declaration , nor communications from a State Party which has not made such a declaration . Article 33 If the Committee receives liable information indicating that a State Party is seriously violating the provisions of this Convention , it may , after consultation with the State Party concerned , request one or more of its members to undertake a visit and report back to it without delay . The Committee shall notify the State Party concerned , in writing , of its intention to organize a visit , indicating the composition of the delegation and the purpose of the visit . The State Party shall answer the Committee within a reasonable time . Upon a s
 ubstantiated request by the State Party , the Committee may decide to postpone or cancel its visit . If the State Party agrees to the visit , the Committee and the State Party concerned shall work together to define the modalities of the visit and the State Party shall provide the Committee with all the facilities needed for the successful completion of the visit . Following its visit , the Committee shall communicate to the State Party concerned its observations and recommendations . Article 34 If the Committee receives information which appears to it to contain well-founded indications that enforced disappearance is being practised on a widespread or systematic basis in the territory under the jurisdiction of a State Party , it may , after seeking from the State Party concerned all relevant information on the situation , urgently bring the matter to the attention of the General Assembly of the United Nations , through the Secretary-General of the United Nations . Article 35 The Co
 mmittee shall have competence solely in respect of enforced disappearances which commenced after the entry into force of this Convention . If a State becomes a party to this Convention after its entry into force , the obligations of that State vis-�-vis the Committee shall relate only to enforced disappearances which commenced after the entry into force of this Convention for the State concerned . Article 36 The Committee shall submit an annual report on its activities under this Convention to the States Parties and to the General Assembly of the United Nations . Before an observation on a State Party is published in the annual report , the State Party concerned shall be informed in advance and shall be given reasonable time to answer . This State Party may request the publication of its comments or observations in the report . Part III Article 37 Nothing in this Convention shall affect any provisions which are more conducive to the protection of all persons from enforced disappear
 ance and which may be contained in : ( a ) The law of a State Party ; ( b ) International law in force for that State . Article 38 This Convention is open for signature by all Member States of the United Nations . This Convention is subject to ratification by all Member States of the United Nations . Instruments of ratification shall be deposited with the Secretary-General of the United Nations . This Convention is open to accession by all Member States of the United Nations . Accession shall be effected by the deposit of an instrument of accession with the Secretary-General . Article 39 This Convention shall enter into force on the thirtieth day after the date of deposit with the Secretary-General of the United Nations of the twentieth instrument of ratification or accession . For each State ratifying or acceding to this Convention after the deposit of the twentieth instrument of ratification or accession , this Convention shall enter into force on the thirtieth day after the date 
 of the deposit of that State 's instrument of ratification or accession . Article 40 The Secretary-General of the United Nations shall notify all States Members of the United Nations and all States which have signed or acceded to this Convention of the following : ( a ) Signatures , ratifications and accessions under article 38 ; ( b ) The date of entry into force of this Convention under article 39. Article 41 The provisions of this Convention shall apply to all parts of federal States without any limitations or exceptions . Article 42 Any dispute between two or more States Parties concerning the interpretation or application of this Convention which cannot be settled through negotiation or by the procedures expressly provided for in this Convention shall , at the request of one of them , be submitted to arbitration . If within six months from the date of the request for arbitration the Parties are unable to agree on the organization of the arbitration , any one of those Parties ma
 y refer the dispute to the International Court of Justice by request in conformity with the Statute of the Court . A State may , at the time of signature or ratification of this Convention or accession thereto , declare that it does not consider itself bound by paragraph 1 of this article . The other States Parties shall not be bound by paragraph 1 of this article with respect to any State Party having made such a declaration . Any State Party having made a declaration in accordance with the provisions of paragraph 2 of this article may at any time withdraw this declaration by notification to the Secretary-General of the United Nations . Article 43 This Convention is without prejudice to the provisions of international humanitarian law , including the obligations of the High Contracting Parties to the four Geneva Conventions of 12 August 1949 and the two Additional Protocols thereto of 8 June 1977 , or to the opportunity available to any State Party to authorize the International Co
 mmittee of the Red Cross to visit places of detention in situations not covered by international humanitarian law . Article 44 Any State Party to this Convention may propose an amendment and file it with the Secretary General of the United Nations . The Secretary-General shall thereupon communicate the proposed amendment to the States Parties to this Convention with a request that they indicate whether they favour a conference of States Parties for the purpose of considering and voting upon the proposal . In the event that within four months from the date of such communication at least one third of the States Parties favour such a conference , the Secretary-General shall convene the conference under the auspices of the United Nations . Any amendment adopted by a majority of two thirds of the States Parties present and voting at the conference shall be submitted by the Secretary-General of the United Nations to all the States Parties for acceptance . An amendment adopted in accordanc
 e with paragraph 1 of this article shall enter into force when two thirds of the States Parties to this Convention have accepted it in accordance with their respective constitutional processes . When amendments enter into force , they shall be binding on those States Parties which have accepted them , other States Parties still being bound by the provisions of this Convention and any earlier amendment which they have accepted . Article 45 This Convention , of which the Arabic , Chinese , English , French , Russian and Spanish texts are equally authentic , shall be deposited with the Secretary-General of the United Nations . The Secretary-General of the United Nations shall transmit certified copies of this Convention to all States referred to in article 38. * Text recommended by the Human Rights Council for adoption by the General Assembly ( see Official Records of the General Assembly , Sixty-first Session , Supplement No. 53 ( A/61/53 ) , part one , chap . Official Records of the 
 General Assembly , Sixty-first Session , Supplement No. 53 ( A/61/53 ) , part one , chap . II . A , resolution 1/1 , annex . 
\ No newline at end of file


[05/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/A/180un_APLC_MSP2_2000_1.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/un/A/180un_APLC_MSP2_2000_1.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/A/180un_APLC_MSP2_2000_1.txt
new file mode 100644
index 0000000..b59a8b4
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/A/180un_APLC_MSP2_2000_1.txt
@@ -0,0 +1,2 @@
+
+THE USE , STOCKPILING , PRODUCTION AND 19 September 2000 AND ON THEIR DESTRUCTION Original : ENGLISH Second Meeting Geneva , 11-15 September 2000 ON THE PROHIBITION OF THE USE , STOCKPILING , PRODUCTION The Final Report of the Second Meeting of the States Parties to the Convention on the Prohibition of the Use , Stockpiling , Production and Transfer of Anti-Personnel Mines and on Their Destruction consists of two parts and five annexes as follows : Part I Organization and Work of the Second Meeting Part II Declaration of the Second Meeting of the States Parties Introduction The Convention on the Prohibition of the Use , Stockpiling , Production and Transfer of Anti-Personnel Mines and on Their Destruction provides in Article 11 , paragraphs 1 and 2 , that : AThe States Parties shall meet regularly in order to consider any matter with regard to the application or implementation of this Convention , including : ( a ) The operation and status of this Convention ; ( b ) Matters arising 
 from the reports submitted under the provisions of this Convention ; ( c ) International cooperation and assistance in accordance with Article 6 ; ( d ) The development of technologies to clear anti-personnel mines ; ( e ) Submissions of States Parties under Article 8 ; and ( f ) Decisions relating to submissions of States Parties as provided for in Article 5@ ; and , Meetings subsequent to the First Meeting of the States Parties Ashall be convened by the Secretary-General of the United Nations annually until the first Review Conference@ . At its fifty-fourth session , the General Assembly of the United Nations in resolution 54/54 B requested the Secretary-General , Ain accordance with Article 11 , paragraph 2 , of the Convention , to undertake the preparations necessary to convene the Second Meeting of the States Parties to the Convention at Geneva , from 11 to 15 September 2000 , and , on behalf of States Parties and according to Article 11 , paragraph 4 , of the Convention , to i
 nvite States not parties to the Convention , as well as the United Nations , other relevant international organizations or institutions , regional organizations , the International Committee of the Red Cross and relevant non-governmental organizations to attend the Meeting as observers@ . To prepare for the Second Meeting , the Standing Committee of Experts ( SCE ) on the General Status and Operation of the Convention , established by the First Meeting of the States Parties , held two meetings , to which all interested States Parties , States not parties to the Convention , as well as the United Nations , other relevant international organizations or institutions , regional organizations , the International Committee of the Red Cross and relevant non-governmental organizations were encouraged to attend . The first meeting of the SCE was held 10-11 January 2000. During the meeting , participants considered a number of issues relating to the organization of the Second Meeting , includ
 ing a draft provisional agenda , a draft programme of work , draft rules of procedure and provisional estimated costs for convening the Second Meeting . No objections were raised in connection with the proposals made with respect to the draft rules of procedure , provisional estimated costs and the venue for the Second Meeting , and it was agreed that they , along with all other conference documents with the exception of reports submitted under Article 7 of the Convention , would be finalized in all six languages of the Convention to be put before the Second Meeting . It was also agreed that the record of work of the five Standing Committees of Experts would be communicated to the Second Meeting in the form of a five-page report prepared by each Committee . The second meeting of the SCE was held 29-30 May 2000. During the meeting , no objections were made with respect to the draft provisional agenda and draft programme of work , and it was agreed that they would be put before the Se
 cond Meeting . In addition , no objections were made with respect to five-page SCE reports serving as the basis for discussion during the informal consultations to be held at the Second Meeting under agenda item 15 ( with the exception of the report of the SCE on the General Status and Operation of the Convention , which would serve as the basis for discussion under agenda item 11 ) . Between the First and Second Meetings of the States Parties , the Standing Committees of Experts received considerable support from the Geneva International Centre for Humanitarian Demining ( GICHD ) . States Parties expressed their appreciation for this assistance and the GICHD=s contribution to the successful operation of the intersessional work programme . In addition , States Parties recognized that the work of the Standing Committees of Experts benefited greatly from the active participation of relevant non-governmental , regional and international organizations . States Parties expressed their gr
 atitude to these organizations for their substantive involvement in the intersessional work programme . The opening of the Second Meeting was preceded by a ceremony at which statements were delivered by the following : Vladimir Petrovsky , Director-General of the United Nations Office at Geneva ; Adolf Ogi , President of Switzerland ; Martine Brunschwig Graf , State Councillor of the Republic and Canton of Geneva ; Alain Vaissade , Mayor of the City of Geneva ; Her Royal Highness , Princess Astrid of Belgium ; Elisabeth Reusse-Decrey , President of the Swiss Campaign against Landmines ; Sir Paul McCartney and Heather Mills . In addition , a presentation was made by 18 landmine survivors from 14 countries . Organization of the Second Meeting The Second Meeting was opened on 11 September 2000 by the President of the First Meeting of the States Parties , the Minister of Foreign Affairs and Cooperation of the Republic of Mozambique , Dr. Leonardo Santos Sim�o . The Second Meeting elect
 ed by acclamation Ambassador Steffen Kongstad of Norway as its President in accordance with rule 7 of the draft rules of procedure . At the opening session , a message addressed to the Second Meeting by the Secretary-General of the United Nations was read by the Director-General of the United Nations Office at Geneva , and statements were made by Jakob Kellenberger , President of the International Committee of the Red Cross , and Jody Williams , Ambassador for the International Campaign to Ban Landmines . At its first plenary meeting on 11 September 2000 , the Second Meeting adopted its agenda as contained in document APLC/MSP.2/2000/L.1 . On the same occasion , the Second Meeting adopted its rules of procedure as contained in document APLC/MSP.2/2000/L.3 , the estimated costs for convening the Second Meeting as contained in document APLC/MSP.2/2000/L.4 , and its programme of work as contained in document APLC/MSP.2/2000/L.2 . Also at its first plenary meeting , representatives from
  Cambodia , Canada , France , Hungary , Mali , Mexico , Mozambique , South Africa , Switzerland and the United Kingdom were elected by acclamation as Vice-Presidents of the Second Meeting . The Meeting unanimously confirmed the nomination of Ambassador Christian Faessler of Switzerland as the Secretary-General of the Meeting . C. Participation and credentials in the Second Meeting Sixty-two States Parties participated in the Meeting : Albania , Argentina , Australia , Austria , Belgium , Benin , Bolivia , Bosnia and Herzegovina , Brazil , Bulgaria , Burkina Faso , Cambodia , Canada , Costa Rica , Croatia , Czech Republic , Denmark , Ecuador , El Salvador , France , Germany , Guatemala , Guinea , Holy See , Honduras , Hungary , Ireland , Italy , Jamaica , Japan , Jordan , Liberia , Liechtenstein , Luxembourg , Madagascar , Malaysia , Mali , Mexico , Monaco , Mozambique , the Netherlands , New Zealand , Nicaragua , Norway , Panama , Peru , Philippines , Portugal , Qatar , Senegal , Sl
 ovakia , Slovenia , South Africa , Spain , Sweden , Switzerland , Thailand , Tunisia , the United Kingdom of Great Britain and Northern Ireland , Venezuela , Yemen and Zimbabwe . Seven States that ratified the Convention , but for which the Convention had not yet entered into force , participated in the Meeting as observers , in accordance with Article 11 , paragraph 4 , of the Convention and rule 1 , paragraph 1 , second sentence , of the rules of procedure of the Meeting : Bangladesh , Colombia , C�te d=Ivoire , Dominican Republic , Gabon , Ghana and Mauritania . A further forty-one States not parties to the Convention participated in the Meeting as observers , in accordance with Article 11 , paragraph 4 , of the Convention and rule 1 , paragraph 1 , second sentence , of the rules of procedure of the Meeting : Afghanistan , Algeria , Angola , Azerbaijan , Belarus , Brunei Darussalam , Burundi , Bhutan , Cameroon , Cape Verde , Chile , China , Cuba , Cyprus , Estonia , Finland , G
 eorgia , Greece , Indonesia , Iraq , Israel , Kenya , Libyan Arab Jamahiriya , Lithuania , Malta , Morocco , Nepal , Oman , Poland , Romania , Saudi Arabia , Singapore , Somalia , Sri Lanka , Sudan , Turkey , Ukraine , United Arab Emirates , Uruguay , Viet Nam and Zambia . Credentials issued by the Head of State or Government or by the Minister for Foreign Affairs or by a person authorized by one of the above , as required by rule 4 of the rules of procedure of the Meeting , or credentials in the form of photocopies or facsimiles of such a document , or credentials in the form of information concerning the appointment of representatives to the Meeting received in the form of letters or notes verbales or facsimiles thereof from embassies , permanent missions to the United Nations or other intergovernmental organizations or other government offices or authorities , were received from all 110 States mentioned in paragraphs 13 to 15 above . The Meeting accepted the credentials of the re
 presentatives of all of the States mentioned in paragraphs 13 to 15 above . In accordance with Article 11 , paragraph 4 , of the Convention and rule 1 , paragraphs 2 and 3 of the rules of procedure , the following international organizations and institutions , regional organizations , entities and non-governmental organizations attended the Meeting as observers : European Union ; International Atomic Energy Agency ( IAEA ) ; International Campaign to Ban Landmines ( ICBL ) ; International Committee of the Red Cross ( ICRC ) ; Organization of American States ( OAS ) ; Sovereign Military Order of Malta ; United Nations Organization : United Nations Secretariat ( Department of Peace-Keeping Operations/United Nations Mine Action Service ( UNMAS ) and Office for the Coordination of Humanitarian Affairs ) , United Nations Children=s Fund ( UNICEF ) , United Nations Development Programme ( UNDP ) , United Nations High Commissioner for Refugees ( UNHCR ) , United Nations Office for Project 
 Services ( UNOPS ) , United Nations Institute for Disarmament Research ( UNIDIR ) , World Health Organization ( WHO ) ; World Bank . In accordance with rule 1.4 , the following organizations attended the Meeting as observers on the invitation of the Meeting : Geneva Foundation , Geneva International Centre for Humanitarian Demining ( GICHD ) , Green Earth Organization , International Development Research Centre ( IDRC ) , League of Arab States , Organisation Internationale de la Francophonie , Organization of the Islamic Conference , PRIO ( International Peace Research Institute , Oslo ) and SOLIDEST . A list of all delegations to the Second Meeting is contained in document APLC/MSP.2/2000/INF.3 . Work of the Second Meeting The Second Meeting held eight plenary meetings from 11 to 15 September 2000. The first three and a half plenary meetings were devoted to the general exchange of views under agenda item 10. Delegations of 34 States Parties , 15 observer States and 7 observer organ
 izations made statements in the general exchange of views , including rights of reply . At the fourth plenary meeting , on 13 September 2000 , the Meeting reviewed the general status and operation of the Convention , expressing satisfaction that 107 States have formally accepted the obligations of the Convention , that 22 States Parties have completed destruction of stockpiled anti-personnel mines and a further 23 States Parties are in the process of stockpile destruction , that the new international norm established by the Convention is taking hold as demonstrated by the behaviour of many States not parties to the Convention , and that approximately US$250 million has been allocated by donors over the past year to address the global landmine problem . The Meeting also expressed satisfaction that efforts to implement the Convention are making a difference , with considerable areas of mined land having been cleared over the past year , with casualty rates having been reduced in sever
 al of the world=s most mine-affected States , and with more and better efforts being undertaken to assist landmine victims . As part of the above-mentioned review , the Meeting reviewed the work of the Standing Committee of Experts on the General Status and Operation of the Convention , as recorded in its report contained in annex IV , and focused its attention on actions recommended by the Committee . At the fifth plenary meeting , on 13 September 2000 , the Meeting considered the submission of requests under Article 5 of the Convention . The President notified the Meeting that he had not been informed that any State wished to make such a request at the Second Meeting . The Meeting took note of this . At the same plenary , the Meeting considered the submission of requests under Article 8 of the Convention . The President notified the Meeting that he had not been informed that any State wished to make such a request at the Second Meeting . The Meeting took note of this . In addition
  , within the framework of the fifth to the seventh plenary meetings , the Meeting held informal consultations on international cooperation and assistance in accordance with Article 6 on the following topics : mine clearance ; victim assistance , socio-economic reintegration and mine awareness ; the destruction of stockpiled anti-personnel mines ; and the development of technologies for mine action . These consultations involved a review of the work of the relevant Standing Committees of Experts , as recorded in their reports contained in annex IV , with a focus on the actions recommended by the Committees . E. Decisions and recommendations At its fifth plenary meeting , on 13 September 2000 , the Meeting considered matters arising from and in the context of reports to be submitted under Article 7 , including consideration of an amendment to the reporting format . The amendment to the format was adopted and is contained in annex III to this report . In addition , the Meeting reviewe
 d the technical ways and means of circulating reports as adopted at the First Meeting , without making any changes to these , with the exception of encouraging States Parties to submit reports electronically and , when submitting an annual update , to highlight changes in relation to earlier reports . Further to the recommendations made by the Standing Committee of Experts on the General Status and Operation of the Convention , the Meeting recognized the continuing importance of the intersessional work programme and , at its fifth plenary meeting , on 13 September 2000 , agreed that the second intersessional work programme would be adjusted according to the President=s paper , which is contained in annex II . Further consultations identified the following States Parties as the Committee Co-Chairs and Rapporteurs until the end of the Third Meeting of the States Parties : - Mine Clearance and Related Technologies : Netherlands and Peru ( Co-Chairs ) ; Germany and Yemen ( Rapporteurs )
  ; - Victim Assistance , Socio-Economic Reintegration and Mine Awareness : Japan and Nicaragua ( Co-Chairs ) ; Canada and Honduras ( Rapporteurs ) ; - Stockpile Destruction : Malaysia and Slovakia ( Co-Chairs ) ; Australia and Croatia ( Rapporteurs ) ; - General Status and Operation of the Convention : Belgium and Zimbabwe ( Co-Chairs ) ; Norway and Thailand ( Rapporteurs ) . States Parties recognized that the work of the Standing Committees would require a high degree of coordination between the Co-Chairs to ensure that their work would facilitate the successful implementation of the Convention . In this context the States Parties established a Coordinating Committee of Co-Chairs , which will meet on an ad hoc basis under the chairmanship of the current President of the Meeting of the States Parties . This Committee will serve to coordinate matters relating to and flowing from the work of the Standing Committees with the work of the Meetings of the States Parties . As appropriate ,
  the Committee can call upon any relevant party to assist in its work , including past Presidents , past Co-Chairs , and representatives of other States Parties and organizations . The Meeting also noted the work undertaken by interested States Parties to establish a sponsorship programme to ensure more widespread representation at meetings of the Convention . States Parties endorsed , and expressed satisfaction with , the work of the Standing Committees of Experts , warmly welcoming the reports of the Standing Committees of Experts , as contained in annex IV . The Meeting was in general agreement with the recommendations made by the Standing Committees of Experts and urged States Parties and all other relevant parties , where appropriate , to act with urgency on these recommendations . At its eighth plenary meeting , on 15 September 2000 , the Meeting agreed that the Third Meeting of the States Parties would be held on 18 to 21 September 2001 in Managua , Nicaragua . At the same pl
 enary , the Meeting adopted the Declaration of the Second Meeting of the States Parties , which is contained in Part II of this report . In addition , the Meeting warmly welcomed the President=s Action Programme , contained in annex V , as a practical means of facilitating implementation of the Convention in accordance with the recommendations made by the Standing Committees of Experts . Documentation A list of documents of the Second Meeting is contained in annex I to this report . Adoption of the Final Report and conclusion of the Second Meeting At its eighth and final plenary meeting , on 15 September 2000 , the Meeting adopted its draft Final Report , contained in document APLC/MSP.2/2000/L.8 . We , the States Parties to the Convention on the Prohibition of the Use , Stockpiling , Production and Transfer of Anti-Personnel Mines and on Their Destruction , along with other States , international organizations and institutions and non-governmental organizations , are gathered in Ge
 neva , Switzerland to reaffirm our unwavering commitment both to the total eradication of anti-personnel mines and to addressing the insidious and inhumane effects of these weapons . We celebrate the ongoing growth in support for the Convention and our satisfaction with the general status and operation of it : over 100 States have formally accepted the obligations of the Convention ; over 20 States Parties have completed destruction of stockpiled anti-personnel mines and a further 23 States Parties are in the process of destroying stockpiles ; the new international norm established by the Convention is taking hold as demonstrated by the behaviour of many States not parties to the Convention ; and approximately US$250 million has been allocated by donors over the past year to address the global landmine problem . We recognize that much work remains . However , we are pleased that our efforts are making a difference : considerable areas of mined land have been cleared over the past ye
 ar ; casualty rates have been reduced in several of the world=s most mine-affected States ; and more and better efforts are being undertaken to assist landmine victims . While we celebrate the success of the Convention , we remain deeply concerned that anti-personnel mines continue to kill , maim and threaten the lives of countless innocent people each day ; that the terror of mines prevents individuals from reclaiming their lives ; and that the lasting impact of these weapons denies communities the opportunity to rebuild long after conflicts have ended . We deplore the continued use of anti-personnel mines . Such acts are contrary to the aims of the Convention and exacerbate the humanitarian problems already caused by the use of these weapons . We call upon all those who continue to use anti-personnel mines , as well as those who develop , produce , otherwise acquire , stockpile , retain and transfer these weapons , to cease now and to join us in the task of eradicating these weapo
 ns . We implore those States that have declared their commitment to the object and purpose of the Convention and that continue to use anti-personnel mines to recognize that this is a clear violation of their solemn commitment . We call upon all States concerned to respect their commitments . We celebrate this Second Meeting of the States Parties . But we recognize that achieving the promise of this unique and important humanitarian instrument rests in continuing to be tireless in our efforts to end the use of anti-personnel mines , to eradicate stockpiles , to cease development , production and transfers of these weapons , to clear mined areas in order to free land from its deadly bondage , to assist victims to reclaim their lives and to prevent new victims . We also recognize that these are common tasks for humanity and therefore call upon all governments and people everywhere to join us in this effort . We call upon those in a position to do so to provide technical and financial a
 ssistance to meet the enormous challenges of mine action , and , whenever relevant , to integrate these efforts into development planning and programming . We call upon those States that have not formally accepted the obligations of the Convention to ratify or accede to it promptly . We call upon all States that are in the process of formally accepting the obligations of the Convention to apply provisionally the terms of the Convention . And we call upon one another as States Parties to effectively implement the Convention and to comply fully with its provisions . We reiterate that , as a community dedicated to seeing an end to the use of anti-personnel mines , our assistance and cooperation will flow primarily to those who have foresworn the use of these weapons forever through adherence to and implementation of the Convention . While we realize that our task is huge , we warmly welcome the substantial progress that has been made during the intersessional work programme and the acc
 omplishments of this programme=s Standing Committees of Experts . We recall that the intersessional work programme was established at the First Meeting of the States Parties to focus and advance the international community=s mine action efforts and to measure progress made in achieving its objectives . We express our satisfaction that the intersessional work programme has lived up to this promise , has assisted in developing a global picture of priorities consistent with the obligations and time-frames contained within the Convention , and has been undertaken in a manner consistent with the Convention=s tradition of inclusivity , partnership , dialogue , openness and practical cooperation . We acknowledge that the progress made during the intersessional work programme was significantly enhanced by the substantive participation of the International Campaign to Ban Landmines and other relevant non-governmental organizations , and by regional and international organizations , including
  the International Committee of the Red Cross . We express our gratitude to these organizations for their important contributions and we thank the Geneva International Centre for Humanitarian Demining for its support of the first intersessional work programme and its commitment to continuing to support future intersessional work . Building upon the accomplishments of the intersessional work programme , including increased participation in the work of the Convention by mine-affected States , we call upon all interested parties to continue to participate in the work of the Standing Committees between now and the next Meeting of the States Parties , which will take place on 18 to 21 September 2001 in Managua , Nicaragua . In reflecting upon our progress and accomplishments , and in considering the work that lies ahead , we reconfirm our conviction to make anti-personnel mines objects of the past , our obligation to assist those who have fallen victim to this terror , and our shared res
 ponsibility to the memories of those whose lives have been lost as a result of the use of these weapons , including those killed as a result of their dedication to helping others by clearing mined areas or providing humanitarian assistance . Symbol Title APLC/MSP.2/2000/L.1 Draft provisional agenda APLC/MSP.2/2000/L.2 Draft programme of work APLC/MSP.2/2000/L.3 Draft rules of procedure APLC/MSP.2/2000/L.4 Estimated costs for convening the Second Meeting of the States Parties to the Convention on the Prohibition of the Use , Stockpiling , Production and Transfer of Anti-Personnel Mines and on Their Destruction APLC/MSP.2/2000/L.5 President=s paper on revisions to the intersessional work programme APLC/MSP.2/2000/L.6 President=s paper on amending the Article 7 reporting format APLC/MSP.2/2000/L.7 President=s Action Programme APLC/MSP.2/2000/L.8 Draft final report of the Second Meeting of the States Parties to the Convention on the Prohibition of the Use , Stockpiling , Production and 
 Transfer of Anti-Personnel Mines and on Their Destruction APLC/MSP.2/2000/SCE1/1 Report of the Standing Committee of Experts on Mine Clearance APLC/MSP.2/2000/SCE2/1 Report of the Standing Committee of Experts on Victim Assistance , Socio-Economic Reintegration and Mine Awareness ( SCE-VA ) APLC/MSP.2/2000/SCE3/1 Report of the Standing Committee of Experts on Stockpile Destruction APLC/MSP.2/2000/SCE4/1 Report of the Standing Committee of Experts on Technologies for Mine Action APLC/MSP.2/2000/SCE5/1 Report of the Standing Committee of Experts on the General Status and Operation of the Convention APLC/MSP.2/2000/1 Final report of the Second Meeting of the States Parties to the Convention on the Prohibition of the Use , Stockpiling , Production and Transfer of Anti-Personnel Mines and on Their Destruction APLC/MSP.2/2000/INF.1 Summary of Article 7 Reports APLC/MSP.2/2000/INF.2 New Zealand - Ottawa Convention Second Meeting of Parties - Anti-personnel mines retained for training APLC/
 MSP.2/2000/INF.3 List of participants APLC/MSP.2/2000/MISC.1 Provisional list of participants PRESIDENT=S PAPER ON REVISIONS TO THE Background At the First Meeting of the States Parties ( FMSP ) , it was decided that the President=s Paper on Intersessional Work ( annex IV of the Final Report ( APLC/MSP.1/1999/1 ) of 20 May 1999 ) would guide the intersessional work . In terms of this decision , informal open-ended Standing Committees of Experts ( SCEs ) were established to focus on the following themes : - Mine clearance ; - Victim assistance , socio-economic reintegration and mine awareness ; - Stockpile destruction ; - Technologies for mine action ; - General status and operation of the Convention . It was further decided that the SCEs could meet on , at least , an annual basis . The SCEs= work programmes for 1999 and 2000 were organized in five periods of meetings in September and December 1999 , January , March and May 2000. This work programme , together with the annual Meeting
  of the States Parties , meant that six periods of meetings were held to promote and facilitate the implementation of mine action in the context of the obligations of the Convention . Although the work programme of the SCEs succeeded in facilitating and supporting the effective functioning of the Convention , the extensive work programme , to a certain degree , constrained participation in , and the efficient organization of , the work of the SCEs . At the 10-11 January 2000 meeting of the SCE on the General Status and Operation of the Convention , the experience with the first meetings of the SCEs was discussed . The need for streamlining the work of the SCEs was identified as well as the importance of broadening the participation in the work of the Committees . At this meeting , it was recommended that the Co-Chairs of this SCE carry out consultations with the objective of formulating concrete suggestions on how the intersessional work programme could be adjusted . In this regard 
 and in addition to various consultations , an informal open-ended consultation session was held at the Geneva International Centre for Humanitarian Demining ( GICHD ) in Geneva on 10 April 2000 , to which States Parties , other States , the International Campaign to Ban Landmines ( ICBL ) and the International Committee of the Red Cross ( ICRC ) were invited . As a result of these consultations , the Co-Chairs of the SCE understood that there was a strong consensus in favour of streamlining the post-Second Meeting of the States Parties ( SMSP ) intersessional work programme . To this end , a series of recommendations were drafted and accepted at the 29-30 May 2000 meeting of the SCE . These recommendations are as follows : Duration of meetings As opposed to the current six periods of meetings , totalling six weeks in duration , it was recommended that only three periods of meetings be held annually , including the Meeting of the States Parties . That is , each SCE shall meet twice b
 etween Meetings of the States Parties , once during an initial week-long session of meetings in November or December 2000 and once during a week-long session of meetings in May 2001. Number of SCEs In the interest of promoting efficiency , directly related themes should be merged into one SCE . It was , therefore , recommended that the SCEs for mine clearance and technologies for mine action be combined into one SCE . Therefore , there would be four SCEs as follows : - Mine clearance and related technologies ( meeting for 1.5 days during each of the two week-long sessions of meetings ) ; - Victim assistance , socio-economic reintegration and mine awareness ( meeting for 1.5 days during each of the two week-long sessions of meetings ) ; - Stockpile destruction ( meeting for 1 day during each of the two week-long sessions of meetings ) ; - General status and operation of the Convention ( meeting for 1 day during each of the two week-long sessions of meetings ) . Language of proceeding
 s To further enhance active participation in the work of the SCEs , it was recommended that those States in a position to do so consider making voluntary contributions to have additional languages made available for the intersessional meetings . Dates of meetings It was recommended that the first of the two week-long sessions of meetings take place 4-8 December 2000. It was further recommended that the second of the two week-long sessions of meetings take place 7-11 May 2001. For example : Role of Co-Chairs It was recommended that Co-Chairs seek , as appropriate , the ongoing support and advice of past Co-Chairs . Name Change It was recommended that what has been to date known as Standing Committees of Experts now be referred to as Standing Committees . Co-Chairs and Rapporteurs As a result of consultations , it was recommended that the following States Parties serve as Co-Chairs and Rapporteurs between the SMSP and the end of the Third Meeting of the States Parties : PRESIDENT=S PA
 PER ON AMENDING THE Background At the January meeting of the Standing Committee of Experts on the General Status and Operation of the Convention , the Article 7 reporting process was discussed . While lessons were learned about difficulties encountered by States Parties in implementing this provision of the Convention , it was the sense of the Co-Chairs that there was general satisfaction with the reporting format accepted at the First Meeting of the States Parties in Maputo . As well , ideas were put forward with respect to possibilities for States Parties to update reports via the Internet . The United Nations should be recognized for the work it has undertaken to develop these ideas . However , concerns were expressed that several States Parties have not submitted reports on time and that of those that have , few have done so electronically . Therefore , it was the sense of the Co-Chairs that while working toward more efficient Internet-based reporting would be a worthwhile mediu
 m-term goal , current efforts with respect to Article 7 reporting should focus on increasing the rate of compliance and ensuring that States in a position to do so submit reports electronically . In addition , as part of the work programme of the Standing Committee of Experts on Victim Assistance , Socio-Economic Reintegration and Mine Awareness , the Victim Assistance Reporting Network Group ( VARNG ) developed a draft reporting form and recommended its acceptance at the 29-31 March 2000 meeting of the SCE on Victim Assistance . The recommendation was not accepted , as States Parties expressed concerns , which included concern that the process may duplicate existing reporting efforts , Areporting fatigue@ , the fact that victim assistance is just one of several Article 6 obligations for which reporting is not an obligation for States Parties , and the difficulties that would be faced by mine-affected States in responding to the proposed form . The Co-Chairs of the SCE on the Genera
 l Status and Operation of the Convention recognized the important role that the SCE on Victim Assistance played in highlighting the need for instruments to indicate the extent to which States Ain a position to do so@ have met their obligations under Article 6 of the Convention , particularly the obligation to Aprovide assistance for the care and rehabilitation , and social and economic reintegration , of mine victims@ . As a possible means of meeting the need highlighted by the SCE on Victim Assistance in a manner that is sensitive to the concerns expressed by States Parties at the March meeting of that Standing Committee , the Co-Chairs of the SCE on the General Status and Operation of the Convention proposed amending the Article 7 reporting format to include an additional form for voluntary reporting on these matters . The proposal put forward by the Co-Chairs was designed to provide an opportunity for States Parties to submit voluntary reports on matters considered important in c
 omplying with obligations under Article 6 , paragraph 3. However , this proposal was also designed to provide States Parties with maximum flexibility in making voluntary reports and to capture , at the discretion of each individual State Party , additional matters pertaining to the implementation of the Convention not covered by formal Article 7 reporting requirements . At the 29-30 May 2000 meeting of the SCE on the General Status and Operation of the Convention , this proposal , in the form of the following recommendations , was accepted : - In order to provide States Parties with the opportunity to report voluntarily on matters pertaining to compliance and implementation not covered by the formal reporting requirements contained in Article 7 , it was recommended that the Article 7 reporting format be amended to include an additional form . ( See attached AForm J : Other relevant matters@ . - It was further recommended that States Parties consider using this form to report on acti
 vities undertaken with respect to Article 6 , in particular to report on assistance provided for the care and rehabilitation , and social and economic reintegration , of mine victims . Form J : Other relevant matters Remark : States Parties may use this form to report voluntarily on other relevant matters , including matters pertaining to compliance and implementation not covered by the formal reporting requirements contained in Article 7. States Parties are encouraged to use this form to report on activities undertaken with respect to Article 6 , and in particular to report on assistance provided for the care and rehabilitation , and social and economic reintegration , of mine victims . State [ Party ] : _______________________________ reporting for time period from _______to________ [ Narrative / reference to other reports ] Report of the Standing Committee of Experts on Mine Clearance to the Second Meeting of the States Parties to the Convention Introduction The Standing Committe
 e of Experts ( SCE ) on Mine Clearance , established in accordance with the decisions and recommendations of the 3-7 May 1999 First Meeting of the States Parties ( FMSP ) , met in Geneva from 13-15 September 1999 and from 27-29 March 2000. At the FMSP , it was agreed in accordance with paragraph 25 of the final report of the FMSP and its annex IV that Mozambique and the United Kingdom would serve as Co-Chairs of the SCE , with the Netherlands and Peru serving as Co-Rapporteurs . Representatives of 69 States , 8 United Nations bodies , the European Commission , the Organization of American States ( OAS ) , the North Atlantic Treaty Organization ( NATO ) , the International Committee of the Red Cross ( ICRC ) , the International Campaign to Ban Landmines ( ICBL ) and of numerous other relevant organizations were registered as participants in either or both of the two meetings . The meetings of the SCE received administrative support from the Geneva International Centre for Humanitaria
 n Demining ( GICHD ) . Matters reviewed by the SCE The SCE considered progress in reviewing and revising international standards for humanitarian mine clearance . The importance of involving all stakeholders and the need for the effective dissemination and implementation of the standards were discussed . The SCE also considered the conditions necessary for establishing a conducive operating environment for mine clearance , and the desirability of agreeing to guidelines for this . The SCE received progress reports on the Study on the Use of Socio-Economic Analysis in Planning and Evaluating Mine Action , which is being undertaken by the GICHD on behalf of the United Nations Development Programme ( UNDP ) , on the Level 1 Survey activities of the Survey Action Center ( SAC ) , and on the impact of mine clearance on peacebuilding and reconstruction . The SCE explored how better planning and the setting of priorities could be assisted by the availability of better quality information , 
 both written portfolios of potential mine action projects and electronic information available on web sites . The United Nations Mine Action Service ( UNMAS ) database , Mine Action Investments , was presented . Ways of improving coordination among stakeholders were examined . Options for improving internal coordination within mine-affected countries were discussed , and ways of better disseminating information internationally were proposed . Recognizing the importance in most cases of developing a sustainable national capacity and the valuable role that the national military could play in certain circumstances , the SCE considered issues related to the use of the military and the improvement of training for enhanced management within national mine action centres . Actions taken or in process on the development of specific tools and instruments in order to assist the implementation of the Convention With the active endorsement of the SCE , and on behalf of UNMAS , the GICHD is revis
 ing international standards for mine clearance , which will be known as the International Standards for Humanitarian Mine Action and contain a glossary of terms and terminology . It was noted that members of the ICBL are reviewing and expanding existing guidelines and principles for good practice in mine clearance . In response to a matter proposed by the first meeting of the SCE , Canada completed the task of developing the UNMAS database , Mine Action Investments , in such a way that donors are able to communicate information on their funding and policies underlying the provision of support for mine action . The database is functioning , though further data would be welcomed . The SCE commended the Portfolio of Mine Action Projects issued by UNMAS and the Compendium Document produced by Handicap International ( HI)/Mine Advisory Group ( MAG)/Norwegian People 's Aid ( NPA ) as useful foundations for identifying worthwhile mine clearance projects to support . UNMAS reported on the I
 nformation Management System for Mine Action ( IMSMA ) , a planning tool for action in mine-affected countries which received widespread interest and support within the SCE . With the encouragement of the SCE , the United Nations has developed guidelines for the use of local militaries in United Nations-supported mine action programmes . The SCE noted the ICBL web site ( www . icbl . org ) and the suggestion that all stakeholders consider contributing to it , including through participation in AE-groups@ . The SCE noted that Landmine Monitor would issue its second report in time for the Second Meeting of the States Parties ( SMSP ) , and that further funding had been requested to complete the report . The SCE commended the GICHD for providing a web site for information related to the work of the SCEs , undertook to provide relevant information and urged all stakeholders to visit and use the SCE portion of the GICHD site . Actions taken or in process to assist in the implementation o
 f the Convention Based upon a statement made by Canada with respect to the compatibility between international standards for mine clearance and obligations under Article 5 of the Convention , the SCE agreed that Convention obligations and international mine clearance standards are not incompatible . With the active encouragement of the SCE , the GICHD is undertaking on behalf of UNDP a Study on the Use of Socio-Economic Analysis in Planning and Evaluating Mine Action . The Survey Action Center has embarked on a programme of Level 1 Survey work in seven countries and has plans for undertaking more . At the suggestion of the SCE , the United Nations agreed to consider making fuller use of the Steering Committee on Mine Action . The SCE called for improved availability of news about mine action , in addition to Landmines magazine and the UNMAS web site . The second Landmine Monitor report would be available for the SMSP and consideration would be given to making available documentation
  from stakeholders ( e. g. national plans and donor criteria ) on web sites . With the active encouragement of the SCE and at the request of UNDP , Cranfield Mine Action is developing , with the support of the United Kingdom , curricula and training materials to improve the skills of mine action managers , and the first course for senior managers is being held from July-September 2000. The SCE noted that subsequent courses would be run at the regional or national level , and would require donor support to assist participation . The SCE commended Nicaragua=s initiative in compiling information about national ( internal ) coordination mechanisms and related best practice , encouraged the continuation of this initiative and suggested that all stakeholders actively consider contributing to it . UNMAS was asked to consider the possibility of more closely involving donors in the annual International Meeting of Mine Action Program Directors and Advisers . Recommendations made by the SCE Th
 e SCE recommended that all stakeholders contribute to the UNMAS/GICHD process of revising international standards for mine action and encouraged broad participation of mine-affected countries . It was further recommended that UNMAS formally advise countries , through their missions in New York , of the opportunity to comment , with a view to UNMAS submitting these revised standards to the United Nations General Assembly at its fifty-fifth session . In addition , it was recommended that national authorities of mine-affected countries , donors and mine action agencies ensure that the new standards are applied . The SCE recommended that consideration be given to the effective dissemination of revised international standards for mine action ( including the translation of the standards ) and to the Mine Action Support Group ( MASG ) playing an active role in disseminating these standards along with other good practice guidelines , like the Bad Honnef Guidelines . The SCE recommended cont
 inued discussions by the Committee on the process of reviewing and revising international standards for mine action , including the implications of applying the standards to the requirements of emergency demining . The SCE recommended that the GICHD , when preparing guidelines for establishing a conducive operating environment for mine clearance , take into account comments and views made by Committee participants , including by the ICBL Mine Action Working Group . In commending the relevance of the Bad Honnef Guidelines , the SCE recommended that HI , MAG and NPA review and expand existing guidance and principles for good practice in humanitarian mine action . The SCE recommended that the UNDP/GICHD present preliminary findings of its Study on the Use of Socio-Economic Analysis in Planning and Evaluating Mine Action to the SMSP and that linkages be further explored between the framework for monitoring and evaluating mine action , as outlined by the International Development Researc
 h Centre ( IDRC ) , and the UNDP/GICHD study . In endorsing the work of the Survey Action Center in its continuing programme of Level 1 Surveys , the SCE recommended that UNMAS/SAC disseminate survey findings and underlined the need for funding to enable additional surveys to take place . The SCE recommended that donors indicate clearly their criteria and requirements for funding in order to help governments and mine action organizations seeking funding to provide proposals which are better tailored to meet donors= funding criteria . The SCE recommended that UNMAS and HI/MAG/NPA consult when updating their respective project portfolios in order to make any appropriate linkages and avoid unnecessary duplication of effort ; that each should identify points of contact ; and that other NGOs should consider contributing to these documents . Recognizing that the UNMAS database Mine Action Investments will be effective only if accurate and substantially complete , the SCE recommended that 
 donors contribute full information and provide regular updates . In addition , the SCE recommended that mine-affected countries check the database and notify UNMAS of any inconsistencies or gaps and that UNMAS consider the possibility of including information on private sector resources in the database . The SCE recommended that UNMAS explore linking relevant countries= and organizations= web sites and , if possible , consider incorporating information about mine affected States= national demining programmes . In addition , it was recommended that mine-affected countries provide texts for inclusion in the United Nations database or details of relevant web sites for linkage to the UNMAS web site . The SCE recommended that UNMAS make more use of the Inter-Agency Steering Committee on Mine Action ( which consists of the relevant United Nations organizations involved in mine action , the ICRC and the ICBL ) and noted the suggestion that the Steering Committee consider possible participa
 tion by regional agencies . The SCE recommended that the OAS and Southern Africa Development Community ( SADC ) share regional experiences in mine action with a view to enhancing coordination and mutual understanding . The SCE recommended that UNMAS identify further ways of disseminating news on mine action , in addition to the regular publication of the Landmines magazine and regular updates of its web site . The SCE recommended further consideration of the prospects for compiling guidelines based on lessons learned in emergency mine action in the event of natural disasters . In addition , the SCE recommended that UNMAS , in the event of a crisis or emergency mine action response , provide to interested parties regular updates on the affected countries= mine situation . The SCE recommended that the IDRC consider extending to other countries the Internet database which has been created for Mozambique . In the light of its discussion on UNMAS guidelines on the use of the military in 
 mine action , the SCE recommended ( a ) that the United Nations Inter-Agency Coordination Group on Mine Action review the way these guidelines are applied on a case-by-case basis ; ( b ) that States Parties and donors consider the possibility of training military trainers of mine-affected countries ; and ( c ) that non-United Nations mine action stakeholders apply the United Nations guidelines in all circumstances where the use of the military is an option . The SCE recommended that donors and implementing agencies consider a multidisciplinary approach to mine action including , where appropriate , other mine action elements in addition to mine clearance , such as victim assistance and mine awareness education . The SCE recommended that the phasing out of direct assistance to national capacities should begin at the appropriate time so as to reduce reliance on foreign assistance ( e. g. the use of expatriate technical assistants ) when adequate local capacities have been developed . 
 The SCE recommended that all relevant actors retain a flexible approach in following mine action guidelines and Committee recommendations . Reference to supporting documents The reports and other documents relating to the two meetings of the SCE in September 1999 and March 2000 may be found on the web site of the GICHD at www . gichd . ch . The UNMAS database Mine Action Investments can be accessed at www . un . org/Depts/dpko/landmines/ . Report of the Standing Committee of Experts on Victim Assistance , Socio-Economic Reintegration and Mine Awareness ( SCE-VA ) to the Second Meeting of the States Parties Introduction The Standing Committee of Experts on Victim Assistance , Socio-Economic Reintegration and Mine Awareness ( SCE-VA ) , established in accordance with the decisions and recommendations of the 3-7 May 1999 First Meeting of the States Parties ( FMSP ) , met in Geneva from 15-17 September 1999 and from 29-31 March 2000. At the First Meeting of States Parties , it was agree
 d in accordance with paragraph 25 of the final report of the FMSP and its annex IV that Mexico and Switzerland would serve as Co-Chairs of the SCE-VA , with Japan and Nicaragua serving as Co-Rapporteurs . Representatives of 43 States Parties , 9 States that signed but have not ratified the Convention , 9 other States , UNMAS , UNDP , the United Nations Children 's Fund ( UNICEF ) , the United Nations High Commissioner for Refugees ( UNHCR ) , the United Nations Department for Disarmament Affairs ( UNDDA ) , the United Nations Institute for Disarmament Research ( UNIDIR ) , the World Health Organization ( WHO ) , OAS , ICRC , ICBL , GICHD and of numerous other relevant organizations were registered as participants in the SCE-VA meetings . The meetings of the SCE-VA received administrative support from the GICHD . The final report of the second meeting of the SCE-VA containing background information , views , opinions , analysis and recommendations can be found at www . gichd . ch . I
 ts reading is highly recommended . Matters reviewed by the SCE-VA At the first SCE-VA meeting , six ANetwork Groups@ were established to facilitate work in the following thematic areas : collection and dissemination of guidelines ; information and data collection ; victim assistance reporting ; mine/unexploded ordnance ( UXO ) awareness ; portfolio of victim assistance programmes ; and donor coordination . The deliberations and outcomes of the Network Groups were discussed at the second meeting of the SCE-VA . Based on a spirit of partnership , shared commitment , and responsibility between civil society and governments , the SCE-VA discussed a comprehensive and integrated approach to victim assistance . The SCE-VA discussed a definition of Alandmine victim@ which includes directly affected individuals , their families , and mine-affected communities . In addition , the SCE-VA discussed Avictim assistance@ as multi-faceted , and requiring a broad range of activities including preven
 tion , emergency medical care , physical and psychological rehabilitation and socio-economic integration . Also considered was victim assistance being defined as an integrated part of assistance to all victims of violence and trauma and persons with disabilities . The SCE-VA discussed , with a view to guaranteeing a long-term sustainable solution , victim assistance being integrated in a broader context of post-conflict reconstruction and development strategies , without losing sight of the directly affected individuals , families and communities which are specifically targeted by the Convention . In particular , the SCE-VA considered the need to have victim assistance integrated into public health , community development , conflict and violence prevention . In addition , the SCE-VA considered the need to include measures relevant to meeting the needs of victims in the principles of humanitarian and development cooperation . During the first meeting of the SCE-VA , Nicaragua and Mex
 ico agreed to facilitate the Network Group on the Collection and Dissemination of Guidelines . The main objective of the Network Group was to try to make existing victim assistance guidelines , including guidelines on socio-economic reintegration and mine awareness , available to all relevant actors . To achieve this objective , the Network Group : called upon all interested actors to contribute in the collection of existing guidelines ; collected all existing victim assistance guidelines received ; discussed how to disseminate the received existing guidelines ; and discussed the need to disseminate a list with the collected guidelines and relevant information to interested parties . During the first meeting of the SCE-VA , Switzerland agreed to facilitate the Network Group on Information and Data Collection . The Network Group discussed a more systematic and reliable data collection and dissemination . Efforts were focused on base line data and quantifying the impact on public heal
 th and reintegration systems , on human and socio-economic development and on the daily life of people and communities . The Network Group emphasized that acquiring victim data should not be an objective in itself but integrated into broader efforts to prevent injuries , assist victims and facilitate better allocation of resources . The Network Group raised issues like national ownership , capacity-building , institutional development and adapting methodologies to reality in the mine-affected countries . The Network Group discussed providing mine-affected countries with methodological support rather than solutions , data collection being handled sensitively , and taking into consideration the impact of data collection on victims . During the first meeting of the SCE-VA , the ICBL and Handicap International agreed to facilitate the Network Group on Victim Assistance Reporting , the purpose of which was to propose a voluntary reporting mechanism . The Network Group raised the issue th
 at , while there is no explicit requirement in the Convention for countries to report on contributions to victim assistance and mine awareness , the Convention requires , in Article 6 , paragraph 3 : AEach State Party in a position to do so shall provide assistance for the care and the rehabilitation , and social and economic reintegration , of mine victims and for mine awareness programmes . @ During the first meeting of the SCE-VA , the ICBL and ICRC agreed to facilitate the Network Group on Mine/UXO Awareness . The Network Group confirmed that appropriate mine/UXO awareness can save lives , that problems faced differ in each situation , but some common elements can be found . During the first meeting of the SCE-VA , the ICBL Working Group on Victim Assistance agreed to facilitate the Network Group on Portfolio of Victim Assistance Programs . The Network Group noted that mention of victim assistance in the Convention has led to a global upsurge of interest in these programmes and 
 that to date there is no global compilation of information on programming . During the first meeting of the SCE-VA , Sweden agreed to facilitate the Network Group on Donor Coordination . The Network Group covered numerous issues of importance ( e. g. recognizing that different States have different needs and understanding the need for both a comprehensive development approach to victim assistance and remembering that victims should not have to wait until macro-level problems are solved before they begin to enjoy a better quality of life ) and identified several ideas for further consideration . Actions taken or in process on the development of specific tools and instruments in order to assist the implementation of the Convention The SCE-VA identified that a strategic , comprehensive and integrated approach to victim assistance is needed and made specific progress in the following areas : ( a ) towards efficient and effective means to monitor Article 6 obligations ; ( b ) in the deve
 lopment , refinement and implementation of various programming and coordination tools ; ( c ) in promoting an information exchange and discussion of various tools for data collection and mine awareness ; ( d ) in promoting various sets of victim assistance and mine awareness guidelines ; and ( e ) in better understanding matters pertaining to victim assistance data collection . The SCE-VA developed a Portfolio of Victim Assistance Programs , the purposes of which are : to raise awareness among governments , donors , and programme implementers on the range of activities that constitute victim assistance ; to promote transparency among all actors in victim assistance ; to highlight needs which have not been addressed because of lack of resources ; and to facilitate contact and information-sharing among actors in victim assistance . Actions taken or in process to assist in the implementation of the Convention Based upon the work of the Network Group on Victim Assistance Reporting , the
  SCE-VA recognized the importance of reporting by all States Parties on support provided for victim assistance and agreed to continue work on effective and efficient means through which States Parties could report on this matter . Recommendations made by the SCE-VA It was recommended that governments establish a mechanism or designate a focal point through which information on victim assistance guidelines could be channelled to appropriate actors in the field and to those elaborating victim assistance , socio-economic reintegration , mine awareness policies or programmes . It was recommended that governments , international and regional organizations , and NGOs interested or involved in victim assistance , socio-economic reintegration and mine awareness , take into account existing guidelines when developing victims assistance programmes and promote a common culture , based on the partnership of governments , international organizations and civil society , about their usefulness in 
 the formulation of said programmes . It was further recommended that ways and means should be found to translate these guidelines into the languages of mine-affected countries . It was recommended that WHO give due attention to enhancing and assessing the quality and performance of surveillance systems for victim assistance in affected countries . It was recommended that governments , international organizations and NGOs share their information at country level with the local bodies ( e. g. inter-sectoral committees , mine action centres , health authorities ) and make the information public . Governments of affected countries should be encouraged to establish victim assistance web sites . It was recommended that new incidents be covered by an adapted health information system producing simple and rapid information as well as by the Information Management System for Mine Action ( IMSMA ) and that further development of the IMSMA incident module be considered . It was recommended tha
 t the WHO integrated surveillance system on victim and trauma be further developed and implemented in a gender-disaggregated form . It was recommended that a platform/clearinghouse for exchange , dissemination and information be established in order to promote transparency , in particular through : the collection and dissemination of standards , methodologies and questionnaires ; the establishment of national platforms and linking them to the international level ; the exchange of research ; and linking existing information about victims . It was further recommended that the GICHD assume this task and develop a test module accessible on its web site . In addition , it was recommended that WHO explore the possibility of engagement in this area . It was recommended that the testing of various tools for victim assistance and mine awareness data collection be promoted . It was recommended that all interested parties continue to work towards efficient and effective means to monitor the im
 plementation of Article 6 , paragraph 3 of the Convention , taking into account the significant work of the Victim Assistance Reporting Network Group . It was recommended that funding be timely , particularly with regards to displaced and returning populations , responsive , appropriate to specific country needs and circumstances , flexible , allowing for the changing reality at the field level , as well as geographically appropriate and coordinated at the donor level . It was recommended that the UNICEF International Guidelines for Mine Awareness be taken into consideration at the outset of programming . It was further recommended that stakeholders should define , together with the mine action coordination body in the mine-affected country , the national standards and methodology for mine action . It was recommended that guidelines be developed for monitoring and evaluating mine/UXO awareness programming and training programmes for mine awareness . It was further recommended that m
 ine/UXO awareness training be integrated into national mine action frameworks . It was recommended that UNICEF continue to provide an open and transparent process ensuring wide participation among mine/UXO awareness actors . It was recommended that UNMAS ensure the integration of the various mine action guidelines into the framework for the development of international standards for mine action . It was recommended that mine/UXO awareness coordination occur whenever possible under a government umbrella , on various levels , including between : ( a ) different implementing agencies involved in mine/UXO awareness programming ; ( b ) the wider mine action community including victim assistance organizations ; and ( c ) the Anon mine action@ humanitarian and development sector , which can bring alternative solutions to mine problems encountered by the communities . It was recommended that mine awareness programmes be implemented in partnership with national mine awareness institutions an
 d actors at the community , regional and national level , from programme identification and assessment , to implementation , monitoring and evaluation . It was further recommended that UNMAS be encouraged to develop a mine/UXO awareness function within the IMSMA to ensure the development of an integrated effective mine action plan and that mine/UXO awareness programming be responsive to both emergency and long-term needs . It was recommended that for the successful implementation of mine action programming , donors/NGOs and all relevant stakeholders ensure that monitoring and evaluation are integral and meaningful parts of all programming . It was recommended that key stakeholders ensure that agencies undertaking mine/UXO awareness activities have a knowledge of and commitment to the existing UNICEF International Guidelines and have ideally a proven capacity and track record of successful programming . It was further recommended that existing mine/UXO awareness agencies should ensur
 e that key documentation is widely available and accessible as learning tools for other agencies . It was recommended that government officials involved in victim assistance receive a copy of the Portfolio as a tool to facilitate contact with other victim assistance actors . It was recommended that States Parties provide resources for the creation and maintenance of a national coordination body , under whose aegis would fall the development of a national Portfolio of Victim Programs . It was recommended that interested parties continue to work together to develop suggestions and recommendations on methods pertaining to more effective donor coordination and long-term resource mobilization with the aim of effectively meeting the immediate and ongoing needs of victims . The focus of this process should be to identify gaps and available resources . It was recommended that , in its future work , the SCE-VA pay due attention to the issues of : ( a ) coordination , with a focus on the clea
 r and precise rationalization of the roles of the major actors in victim assistance ; ( b ) identifying gaps in terms of financial , technical and other resources needed for victim assistance ; and ( c ) measuring progress towards implementation of the Convention , particularly with a focus on examining victim assistance reporting mechanisms . It was recommended that the future work of the SCE-VA should be divided into two types of activities : ( a ) operationalization of work already begun ( e. g. following through on the most salient recommendations of the Network Groups ; setting precise goals ; and collecting and disseminating documents from the SCE-VA ) ; and ( b ) analysis of new themes ( e. g. creating an inventory of existing policies , studies , surveys and lessons learned ; considering links between victim assistance and mine awareness and long-term strategies for sustainable development ; and increasing the participation in mine-affected countries of civil society and , p
 articularly , associations of mine victims and/or persons with disabilities ) . Reference to supporting documents As a result of the work of the SCE-VA=s Network Groups , six papers were produced containing the views of these groups and the proposals put forward by them to the SCE-VA=s March 2000 meeting . These papers can be found at : www . gichd . ch . Report of the Standing Committee of Experts on Stockpile Destruction to the Second Meeting of the States Parties to the Convention Introduction The Standing Committee of Experts ( SCE ) on Stockpile Destruction , established in accordance with the decisions and recommendations of the 3-7 May 1999 First Meeting of the States Parties ( FMSP ) , met in Geneva from 9-10 December 1999 and from 22-23 May 2000. At the FMSP , it was agreed , in accordance with paragraph 25 of the final report of the FMSP and its annex IV , that Hungary and Mali would serve as Co-Chairs of the SCE , with Malaysia and Slovakia serving as Co-Rapporteurs . Rep
 resentatives of 47 States , United Nations bodies , the ICRC , ICBL and numerous other relevant organizations were registered as participants in either or both of the two meetings . The meetings of the SCE received administrative support from the GICHD . Matters reviewed by the SCE The SCE solicited the views of delegations with respect to the following thematic areas : stockpile destruction as an integral part of mine action ; allocation of resources - technologies and constraints ; case studies ; stockpile destruction as preventive mine action ; cooperative structures for stockpile destruction ; and the way ahead . The SCE noted its appreciation of the work of the following moderators : Ret . Gen. Gordon M. Reay , Department of Foreign Affairs and International Trade , Canada ; Patrick Blagden , GICHD ; Col. Paul Power , Australian Defense Forces ; Steve Goose , Human Rights Watch ; and Adrian Wilkinson , Mine Action Consultant , UNDP . The SCE considered a number of practical iss
 ues with a view to highlighting the importance of the core objective of the SCE , namely to facilitate a swift and dramatic reduction in the number of stockpiled anti-personnel mines globally , including the following : ensuring political priority for stockpile destruction ; obligations and rights of countries under Article 4 of the Convention ; merits and constraints of various methods and technologies for destruction as experienced by individual countries ; the role of the military and private sector in stockpile destruction ; logistical , technical and financial considerations ; possible alternatives to the current methods of stockpile destruction ; planning and implementation of the process leading up to the actual destruction of stockpiles ; financial and technical assistanceBbilateral , multilateral and regional approaches to stockpile destruction ; compilation of a database of industrial/research capacities and capabilities ; possible mechanisms for monitoring and evaluating 
 Article 7 reports ; assessing overall progress with regard to stockpile destruction ; progress report on global stockpiles and their destruction ; flow of information on available technologies , costs and environmental impact ; modalities of transfer and storage of foreign stockpiles ; avoiding competition among the various branches and actors of mine action ; engaging the media and the public at large in the process of stockpile destruction ; need for accounting and certification procedures ; compilation of databases on donors , recipients , needs , methods , options , companies , experts ; and possible mechanisms for engaging non-States parties in reducing their stockpiles . It was expected that discussion on all the above matters would continue in the intersessional work programme . As a result of the SCE=s deliberations , it was accepted that stockpile destruction is an integral part , in effect the Afifth pillar@ , of mine action and that , accordingly , compliance with Article
  4 obligations should receive high political priority . It was also stressed , however , that this does not suggest unwarranted competition among the different branches of mine action . At the FMSP , participants emphasized the need for the rapid universalization of , adherence to and compliance with the Convention and , in parallel , the importance of prompt and strict compliance with the obligations as stipulated in Article 4. Accordingly , States were urged to comply with their reporting obligations under Article 7 as a way to facilitate future cooperation between prospective donor States and those requesting assistance in carrying out this important task . In the course of its intersessional work , the Committee sought ways to ensure that the capabilities and capacities of prospective donor States become fully compatible with the needs of States requesting assistance . In addition , the SCE reviewed a wide range of technical options for stockpile destruction , which were introdu
 ced in the course of the sessions for consideration . Actions taken or in process on the development of specific tools and instruments in order to assist the implementation of the Convention A web site related to stockpile destruction is expected to be introduced by UNMAS and Canada by September 2000 and States were encouraged to contribute to this effort . Proposed UNDP guidelines on stockpile destruction will be made available on the web as well . A list of companies , experts and technologies related to stockpile destruction is also expected to be introduced in the near future . A cumulative list of companies , experts and technologies could provide a necessary link between the Adonor@ and Arecipient@ countries when designing future cooperative structures . Regarding monitoring and verification requirements , a number of presentations have already been prepared , which serve as good examples . It was acknowledged that case study presentations are a useful tool for maintaining pol
 itical enthusiasm and that a standardized format for the case study presentations could be identified vis-�-vis the ones that have been presented up to date . The submission of ( annual ) national progress reports on stockpile destruction was not a matter of consensus at this point of time . It was observed that there are a number of relevant questions not included in the Article 7 reporting format ; therefore , a proposal for a separate questionnaire on stockpile destruction was entertained . The important role the ICBL Landmine Monitor plays in reporting on the global stockpile destruction process and contributing to greater transparency on this issue was highly appreciated . Actions taken or in process to assist in the implementation of the Convention The need for additional resources to be secured for stockpile destruction projects was stressed . To this end , the sessions considered various bilateral , multilateral and regional approaches as possible components for establishin
 g cooperative structures for stockpile destruction projects . The initiative launched by Canada and Ukraine was cited in this regard as a useful example for bilateral cooperation , while the establishment of a Partnership for Peace ( PfP ) Trust Fund for stockpile destruction within the framework of the North Atlantic Treaty Organization 's Euro-Atlantic Partnership Council ( NATO/EAPC ) was singled out as a promising multilateral/regional project with a view to facilitating broad compliance with Article 4 obligations . More than a dozen countries provided insights into their experience with stockpile destruction and representatives offered their respective views with regard to the merits and constraints of various methods of destruction as experienced by individual countries . Financial , technical , social and environmental considerations were also discussed at length , and emphasis was placed on the need for careful planning and implementation of the process leading up to the act
 ual destruction of stockpiles . The importance of engaging the media and the public at large in the process of stockpile destruction was also highlighted by various delegations . The SCE compared the converging or diverging approaches taken by the military and civil sectors in the actions needed for eliminating stockpiled anti-personnel mines . In most cases , it was argued , stockpile destruction in military facilities leads to a substantial reduction of incurred expenses and entails the more efficient utilization of already existing resources . Transparency in this process is also of critical importance , a fact about which the armed forces should be educated . An important role should be assigned to players outside the military to ensure maximum transparency in the process of stockpile destruction ( governmental agencies , mass media , diplomatic corps , etc. ) . The possible roles of UNMAS and UNDP in stockpile destruction projects were also discussed . With its 137 regional cen
 tres worldwide , UNDP could facilitate various bilateral or multilateral arrangements with a view to promoting information exchange and technical/financial cooperation in this field . Although the United Nations agencies are traditionally involved in humanitarian demining activities , the possibility of expanding their activities to facilitate stockpile destruction should not be excluded . The possible modalities of such United Nations involvement , e. g. transparency and lessons learned and shared , and financial assistance , were also discussed . Retention of stockpiled anti-personnel mines for training and development purposes under Article 3 was also discussed at length , although the Co-Chairs made the determination that the issue should be referred to the SCE on the General Status and Operation of the Convention for further deliberation . Generally , it was agreed that examination of all options and factors should lead to the destruction of global stocks in the fastest , most 
 cost-effective way and with the least harmful effects to the environment . It was also emphasized that the possibility of adverse environmental impacts brought about by the destruction process should not be used as an excuse for inaction in meeting Article 4 obligations . The Committee agreed that , at the SMSP , the urgency and importance of stockpile destruction should be affirmed . Recommendations made by the SCE It was recommended that those countries that have completed the destruction of their stockpiles should make their expertise available to those in need , because in many instances countries lack the technical experience , industrial capacities or know-how to carry out obligations stipulated in Article 4 of the Convention . It was recommended that States be encouraged to contribute to the questionnaire presented by Canada in support of the development of an UNMAS web site on stockpile destruction . With regard to a complementary database to the Article 7 reports , it was r
 ecommended that the SCE Co-Chairs , in cooperation with interested parties , work to develop a questionnaire that would help collect information on needs and assistance offers expressed by non-States Parties . It was recommended that the SCE Co-Chairs develop specific language on recommendations concerning bilateral and multilateral/regional cooperative structures for stockpile destruction . It was recommended that an examination of all options and factors should lead to the destruction of global stocks in the fastest , most cost-effective way and with the least harmful effects to the environment . Report of the Standing Committee of Experts on Technologies for Mine Action to the Second Meeting of the States Parties Introduction The Standing Committee of Experts on Technologies for Mine Action ( SCETMA ) , established in accordance with the decisions and recommendations of the First Meeting of the States Parties ( FMSP ) , which was held from 3-7 May 1999 in Maputo , Mozambique , me
 t in Geneva on 13-14 December 1999 and on 24-25 May 2000. In accordance with paragraph 25 of the final report of the FMSP , it was agreed that Cambodia and France would serve as Co-Chairs of the SCE , with Germany and Yemen serving as Co-Rapporteurs . Representatives of 34 States that have ratified the Convention , 7 States that signed but have not ratified the Convention , 9 other States , the ICRC , the ICBL and 8 regional and international organizations , including the United Nations , and numerous other relevant organizations , including national mine action centres , universities , research centres and companies , were registered as participants in the SCE meetings . The meetings of the SCE received administrative support from the GICHD . Matters reviewed by the SCE The SCETMA benefited from a pragmatic and clear mandate as well as a lively and dedicated participation by experts whose diversity and complementarity found full expression in the course of the debates . The SCETMA 
 had been mandated to conduct a comparative survey of the needs of the end-users and of the technologies available to them or in development . The Committee fulfilled its mandate by taking into account the constraints which stand in the way of providing end-users with appropriate technologies , as well as the new standards required by the technological progress of mine action . This framework provided ample opportunity to field practitioners , programme managers , specialists from academia and research establishments as well as military and industrial decision-makers , to interact and to exchange information . This provided a better understanding of the adequacy of technologies currently or soon to be available for mine action , of the new international standards that ought to be implemented , and of the most promising progress in technology research . The Committee=s work was enhanced by useful contributions by individuals responsible for mine action field work , including individua
 ls from Mine Action Centres ( e. g. C-MAC ( Cambodia ) , CROMAC , IND-Mozambique , NCHD-Chad , Monitoring , Evaluation and Training Agency ( META ) and MAP-Afghanistan , CND-Nicaragua ) , international organizations ( e. g. UNMAS and UNDP ) and non-governmental organizations ( e. g. ICBL , Handicap International , Mines Advisory Group , Norwegian People 's Aid ) and GICHD . The military establishments of donor countries and affected States also participated . The scientific sector ( e. g. RMA-Belgium , ERA-United Kingdom , JRC-European Commission ) and the industrial sector ( e. g. Mechem-South Africa ) provided an overview of the ways used to attempt to adapt existing technologies . End-users and the research and development actors , academics ( e. g. University of Western Australia , Cranfield University , James Madison University , the European Commission Joint Research Centre ( JRC ) ) discussed their own efforts at rationalizing mine action in the face of the diversity of hurdl
 es it has to overcome . Actions taken or in process on the development of specific tools and instruments in order to assist the implementation of the Convention The SCETMA confirmed that taken separately neither manual demining , nor mine detecting dogs , nor mechanical equipment , holds the key to the landmine problem . The key lies in the so called Atool box@ approach , wherein these techniques and technologies are complementary , and need to be used in different combinations after a careful review of the conditions and environment of each minefield or mine action programme . The SCETMA emphasized different elements of the tool box used by different mine action centres , from Cambodia to Afghanistan , Mozambique , Chad , Nicaragua or Croatia . The diverse elements presented reflect varying sets of constraints , ranging from human , geographical and climatic conditions , to financial , organizational or political limitations . Mine action centres are well placed to give proper cons
 ideration , for instance , to the variety of mines in place in the same area , or to the demands made on the mobility and modularity of teams and equipment , in consideration of different conditions in the field , from waterlogged to arid soils , from sandy to rocky terrains , etc. The SCETMA noted that while the tool box approach is widely accepted , there is room for a variety of opinions on other aspects of mine action in the field . With respect to mine detection dogs ( MDD ) for example , the experience of deminers in Afghanistan shows that MDDs are fast and effective , provided they are assigned appropriate tasks in area reduction or clearance in low-density mined areas . However , deminers in Kosovo pointed out that it is indispensable to adopt an accredition procedure in order to guarantee , before and during clearance operations , the quality of dogs supplied . The University of Western Australia suggested Adouble blind tests@ for dogs and supervisors . Universal interest i
 n the MDD tool , and lingering doubts on its proper use , have led UNMAS and the GICHD to launch no less than eight studies to cover every aspect of the MDD technique in the course of the next few years . The SCETMA noted different , but not irreconcilable , approaches to the use of mechanical equipment . Suppliers of heavy equipment stressed its effectiveness if assigned appropriate tasks , from road clearance to vegetation-cutting and other preparations of the field . Practitioners in the field underlined the need for multi-purpose platforms and increased sustainability of the equipment purchased or leased , and for improved procedures for testing in the field . All were in agreement on the necessity to integrate the selection of appropriate mechanical equipment in the early stages of planning for demining . The SCETMA exposed concern at the multiplication of databases and information technologies available , and the need to promote compatibility and interconnectednesss . Field wo
 rkers insisted on the need to preserve an end-user friendly approach . Progress made by the GICHD Information Management System for Mine Action ( IMSMA ) was applauded - a system which between the two meetings of the SCETMA was developed and tested in a growing list of mine-affected countries . The SCETMA acknowledged the importance of the current exhaustive review of United Nations norms and standards . End-users will systematically be associated with this process . The new United Nations standards will also be compatible with those of the International Organization for Standardization ( ISO ) . However , concern was raised regarding the need to identify who will assume the responsibility for implementing these standards and verifying their correct implementation . The SCETMA noted that stakeholders in mine action now realize that the time-lines of scientific innovation and progress and those of demining requirements are not necessarily the same . It was undersand that States most 
 active in research and development of new demining technologies do so primarily for military reasons , with less attention to the particular needs of humanitarian demining . This bias , however , was not universal : international ( International Atomic Energy Agency ( IAEA ) ) , regional ( European Commission-JRC ) , and national institutions do have specific programmes or projects aimed at humanitarian demining . All stakeholders agreed to stress the importance of a cooperative approach among researchers , developers and deminers . Actions taken or in process to assist in the implementation of the Convention To faciliate the implementation of the Convention , the SCETMA concluded that there is an overall need to nurture a dual approach in the field of mine action technology , namely : the development of simple , end-user friendly technologies based on improving existing material , as being essential to ease the burden of deminers and to speed up ongoing programmes ; and the search 
 for high-tech innovations , as being critical to save lives , time and money in the longer term . Recommendations made by the SCE With the view that cooperation between mine action stakeholders must be strengthened in several ways and at several levels , the SCETMA recommended : The exchange of relevant information between end-users , in particular between mine action centres ; The systematic field testing of new technologies in order ( a ) to provide researchers with a better appreciation of the improvements needed and a better access to data acquired by deminers and ( b ) to increase deminers ' awareness and acceptance of new technologies ; The facilitation of the transfer of equipment through the adoption and implementation of Ademining friendly regulations@ ; The development of integrated databases such as IMSMA and fostering of the compatibility and interconnection of existing databases ; Facilitating access to national resources in terms of conventional and digitalized maps , 
 surveys and other relevant documentation concerning mined areas , subject to national regulations and considerations of national security ; The development of software , using information available in databases , with a view to assist those responsible for mine action in their choice of adapted technologies ; Setting up of a network of probing facilities and international testing centres ; Defining common standards for field testing ; and Defining mechanisms and procedures to allocate new technologies to mine action teams ( e. g. with respect to its discussion on UNMAS policy on the use of militaries in mine action , the SCETMA recommended ( a ) that the United Nations Inter-Agency Coordination Group on Mine Action review the way these guidelines are applied on a case-by-case basis , ( b ) that States Parties and donors consider the possibility of training military trainers of mine-affected countries , and ( c ) that non-United Nations mine action stakeholders be urged to apply Unit
 ed Nations guidelines in all circumstances where the use of the military is an option ) . Report of the Standing Committee of Experts on the General Status 

<TRUNCATED>

[51/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
merge from bgalitsky's own git repo


Project: http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/repo
Commit: http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/commit/1f97041b
Tree: http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/tree/1f97041b
Diff: http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/diff/1f97041b

Branch: refs/heads/master
Commit: 1f97041ba33317481b0f2083f7316a312b8d8b52
Parents: 9aa270c
Author: Boris Galitsky <bg...@hotmail.com>
Authored: Wed Nov 16 10:10:18 2016 -0800
Committer: Boris Galitsky <bg...@hotmail.com>
Committed: Wed Nov 16 10:10:18 2016 -0800

----------------------------------------------------------------------
 opennlp-similarity/README.md                    |   157 +
 opennlp-similarity/README.txt                   |   138 +
 .../lib/edu.mit.jverbnet-1.2.0.jar              |   Bin 0 -> 168444 bytes
 opennlp-similarity/lib/ejml-0.23.jar            |   Bin 0 -> 211938 bytes
 opennlp-similarity/lib/javax.json.jar           |   Bin 0 -> 85147 bytes
 opennlp-similarity/lib/joda-time.jar            |   Bin 0 -> 570478 bytes
 opennlp-similarity/lib/jollyday.jar             |   Bin 0 -> 200993 bytes
 opennlp-similarity/lib/xom.jar                  |   Bin 0 -> 313253 bytes
 opennlp-similarity/src/main/java/com.zip        |   Bin 0 -> 47523 bytes
 .../tools/apps/object_dedup/DedupResult.java    |    31 +
 .../object_dedup/SimilarityAccessorBase.java    |   739 +
 .../main/java/opennlp/tools/doc_classifier.zip  |   Bin 0 -> 15046 bytes
 .../ClassifierTrainingSetIndexer.java           |   266 +
 .../tools/doc_classifier/DocClassifier.java     |   306 +
 ...assifierTrainingSetMultilingualExtender.java |   284 +
 .../DocClassifierTrainingSetVerifier.java       |   163 +
 .../enron_email_recognizer/EmailNormalizer.java |   100 +
 .../EmailTrainingSetFormer.java                 |    67 +
 .../opennlp/tools/fca/BasicLevelMetrics.java    |   676 +
 .../java/opennlp/tools/fca/ConceptLattice.java  |   298 +
 .../java/opennlp/tools/fca/FcaConverter.java    |    72 +
 .../main/java/opennlp/tools/fca/FcaReader.java  |    95 +
 .../main/java/opennlp/tools/fca/FcaWriter.java  |   131 +
 .../java/opennlp/tools/fca/FormalConcept.java   |   152 +
 .../main/java/opennlp/tools/fca/Measures.java   |   147 +
 .../opennlp/tools/fca/RandomNoiseGenerator.java |    63 +
 .../opennlp/tools/jsmlearning/JSMDecision.java  |    78 +
 .../jsmlearning/JSMLearnerOnLatticeBase.java    |   338 +
 .../JSMLearnerOnLatticeWithAbduction.java       |    88 +
 .../JSMLearnerOnLatticeWithDeduction.java       |   243 +
 .../parse_thicket/ParseCorefBuilderWithNER.java |   156 +
 .../tools/parse_thicket/VerbNetProcessor.java   |   267 +
 .../external_rst/ExternalRSTImporter.java       |   264 +
 .../external_rst/MatcherExternalRST.java        |    96 +
 .../PT2ThicketPhraseBuilderExtrnlRST.java       |   115 +
 .../ParseCorefBuilderWithNERandRST.java         |   268 +
 .../ParseThicketWithDiscourseTree.java          |   284 +
 .../parse_thicket/external_rst/RstNode.java     |   119 +
 .../kernel_interface/BracesProcessor.java       |   197 +
 .../DescriptiveParagraphFromDocExtractor.java   |   168 +
 .../TreeKernelBasedClassifier.java              |   272 +
 .../TreeKernelBasedClassifierMultiplePara.java  |   200 +
 ...reeKernelBasedClassifierOfDiscourseTree.java |   326 +
 .../style_classif/TSNE_ImporterProcessor.java   |    94 +
 .../matching/FrameQueryBasedIExtractor.java     |   153 +
 .../matching/GeneralizationResult.java          |    66 +
 .../matching/LemmaGeneralizer.java              |    94 +
 .../tools/parse_thicket/matching/MyMatcher.java |   126 +
 .../matching/NERPhraseGeneralizer.java          |   271 +
 .../matching/ParseTreeNodeGeneralizer.java      |    68 +
 .../matching/PartOfSpeechGeneralizer.java       |    77 +
 .../matching/PersonalInformationExtractor.java  |   159 +
 .../matching/PhraseGeneralizer.java             |   268 +
 .../matching/PhraseGroupGeneralizer.java        |   128 +
 .../opinion_processor/AbstractEngineRunner.java |   120 +
 .../DefaultSentimentProcessor.java              |   523 +
 .../EntityExtractionResult.java                 |   158 +
 .../ExpressionSentimentAnalyzer.java            |    56 +
 .../LinguisticPhraseManager.java                |   591 +
 .../opinion_processor/NamedEntityExtractor.java |   185 +
 .../opinion_processor/PersonExtractor.java      |    96 +
 .../SentencePhraseGivenAWordGetter.java         |   107 +
 .../SentimentCoreAnnotations.java               |    41 +
 .../opinion_processor/StopList.java             |   401 +
 .../TopicAsOpinionMinerRunner.java              |   117 +
 .../opinion_processor/TopicPhraseExtractor.java |   212 +
 .../opinion_processor/TwitterEngineRunner.java  |   157 +
 .../opinion_processor/TwitterFilter.java        |   165 +
 .../opinion_processor/YouTubeMiner.java         |    97 +
 .../opinion_processor/YouTubeMinerResult.java   |    38 +
 .../LinguisticPatternStructure.java             |   192 +
 .../PatternStructureWriter.java                 |    59 +
 ...KernelBasedRecognizerOfRequest_Response.java |    94 +
 .../YahooAnswersTrainingSetCreator.java         |   118 +
 .../apps/BingRelatedSpellingQueryRunner.java    |   105 +
 .../apps/GoogleAutoCompleteQueryRunner.java     |   124 +
 .../similarity/apps/SentenceTranslate.java.txt  |   212 +
 .../tools/similarity/apps/solr/Comment.java     |   118 +
 .../tools/similarity/apps/solr/CommentsRel.java |   139 +
 .../apps/solr/QueryExpansionRequestHandler.java |   105 +
 ...earchResultsReRankerStanfRequestHandler.java |   306 +
 .../similarity/apps/solr/WordDocBuilder.java    |   270 +
 .../apps/solr/WordDocBuilderEndNotes.java       |   223 +
 .../WordDocBuilderSingleImageSearchCall.java    |   174 +
 .../apps/taxo_builder/CsvAdapter.java           |    74 +
 .../taxo_builder/DomainTaxonomyExtender.java    |   230 +
 .../java/opennlp/tools/stemmer/PStemmer.java    |   521 +
 .../tools/word2vec/W2VDistanceMeasurer.java     |   124 +
 .../test/java/opennlp/tools/fca/FCATest.java    |    79 +
 .../external_rst/ExternalRSTImporterTest.java   |    47 +
 .../JSMLearnerOnLatticeTest.java                |   317 +
 .../pattern_structure/PatternStructureTest.java |   189 +
 .../pattern_structure/PhraseTest.java           |   171 +
 .../test/resources/external_rst/resInput.txt    |    62 +
 .../src/test/resources/fca/sports.cxt           |    55 +
 .../src/test/resources/new_vn.zip               |   Bin 0 -> 389230 bytes
 .../resources/style_recognizer/all-tsne2.txt    |  1051 ++
 .../txt/Bib1/27Bib1Amos_Prophecy_12_EN.txt.txt  |     2 +
 .../txt/Bib1/28Bib1Amos_Prophecy_89_EN.txt.txt  |     2 +
 .../txt/Bib1/29Bib1Ezra_History_12_EN.txt.txt   |     2 +
 .../txt/Bib1/30Bib1Joel_Prophecy_12_EN.txt.txt  |     2 +
 .../txt/Bib1/31Bib1Ruth_History_12_EN.txt.txt   |     2 +
 .../txt/Bib2/32Bib2John_Gospel_12_EN.txt.txt    |     2 +
 .../txt/Bib2/33Bib2Jude_Epistle_1_EN.txt.txt    |     2 +
 .../txt/Bib2/34Bib2Luke_Acts_12_EN.txt.txt      |     2 +
 .../txt/Bib2/35Bib2Luke_Acts_2728_EN.txt.txt    |     2 +
 .../txt/Bib2/36Bib2Luke_Gospel_12_EN.txt.txt    |     2 +
 .../txt/Bib2/37Bib2Mark_Gospel_12_EN.txt.txt    |     2 +
 .../txt/Corp/38CorpApple_Environment_EN.txt.txt |     2 +
 .../txt/Corp/39CorpHandM_Quality_EN.txt.txt     |     2 +
 .../Corp/40CorpHandM_Responsibility_EN.txt.txt  |     2 +
 .../txt/Corp/41CorpHP_Objectives_EN.txt.txt     |     2 +
 .../Corp/42CorpJaguar_Sustainability_EN.txt.txt |     2 +
 .../txt/Corp/43CorpZara_Environment_EN.txt.txt  |     2 +
 .../txt/Fict/44FictAndersenH_Mermaid_EN.txt.txt |     2 +
 .../txt/Fict/45FictAndersenH_Ugly_EN.txt.txt    |     2 +
 .../txt/Fict/46FictBalzacH_Goriot_Ia_EN.txt.txt |     2 +
 .../txt/Fict/47FictBalzacH_Goriot_Ib_EN.txt.txt |     2 +
 .../Fict/48FictBronteC_JaneEyre_33_EN.txt.txt   |     2 +
 .../Fict/49FictBronteC_JaneEyre_6_EN.txt.txt    |     2 +
 .../Fict/50FictCarrollL_Wonderland_2_EN.txt.txt |     2 +
 .../Fict/51FictCarrollL_Wonderland_5_EN.txt.txt |     2 +
 .../52FictDostoyevskyF_CrimePun_ep_EN.txt.txt   |     2 +
 .../53FictDostoyevskyF_CrimePun_II2_EN.txt.txt  |     2 +
 .../txt/Fict/54FictDumasAp_3Musk_11_EN.txt.txt  |     2 +
 .../txt/Fict/55FictDumasAp_3Musk_2_EN.txt.txt   |     2 +
 .../Fict/56FictFlaubertG_Bovary_I5_EN.txt.txt   |     2 +
 .../Fict/57FictFlaubertG_Bovary_III3_EN.txt.txt |     2 +
 .../Fict/58FictFlaubertG_Salammbo_11_EN.txt.txt |     2 +
 .../Fict/59FictFlaubertG_Salammbo_2_EN.txt.txt  |     2 +
 .../Fict/60FictGrimmJ_Bremusicians_EN.txt.txt   |     2 +
 .../Fict/61FictGrimmJ_HanselGretel_EN.txt.txt   |     2 +
 .../Fict/62FictHugoV_Hunchback_II5_EN.txt.txt   |     2 +
 .../Fict/63FictHugoV_Hunchback_IV3_EN.txt.txt   |     2 +
 .../txt/Fict/64FictPoeE_Purloined_EN.txt.txt    |     2 +
 .../txt/Fict/65FictPoeE_Usher_EN.txt.txt        |     2 +
 .../Fict/66FictTolstoyL_AnnaK_I10_EN.txt.txt    |     2 +
 .../Fict/67FictTolstoyL_AnnaK_I11_EN.txt.txt    |     2 +
 .../Fict/68FictTolstoyL_WarPeace_I16_EN.txt.txt |     2 +
 .../Fict/69FictTolstoyL_WarPeace_I2_EN.txt.txt  |     2 +
 .../txt/Fict/70FictWildeO_Dorian_18_EN.txt.txt  |     2 +
 .../txt/Fict/71FictWildeO_Dorian_8_EN.txt.txt   |     2 +
 .../Fict/72FictZolaE_Germinal_II4_EN.txt.txt    |     2 +
 .../Fict/73FictZolaE_Germinal_VII2_EN.txt.txt   |     2 +
 .../txt/Marx/74MarxMarxK_ComMan_01_EN.txt.txt   |     2 +
 .../txt/Marx/75MarxMarxK_ComMan_24_EN.txt.txt   |     2 +
 .../txt/Marx/76MarxMarxK_Feuerbach_EN.txt.txt   |     2 +
 .../txt/Marx/77MarxMarxK_GothaProg_EN.txt.txt   |     2 +
 .../txt/Marx/78MarxMarxK_JewishQ_EN.txt.txt     |     2 +
 ...k_international_story_0,,631931,00.html.txt" |     2 +
 ....bbc.co.uk_2_hi_middle_east_5136988.stm.txt" |     2 +
 ...bbc.co.uk_2_hi_asia-pacific_2181151.stm.txt" |     2 +
 ...ews.bbc.co.uk_2_hi_business_2235352.stm.txt" |     2 +
 ..._money.guardian.co.uk_news__0,,,00.html.txt" |     2 +
 ...n.co.uk_Match_Report_0,,1536875,00.html.txt" |     2 +
 ...bbc.co.uk_2_hi_asia-pacific_5133220.stm.txt" |     2 +
 ..._news.bbc.co.uk_2_hi_europe_4263426.stm.txt" |     2 +
 ...ews.bbc.co.uk_2_hi_business_2769931.stm.txt" |     2 +
 ...o.uk_2_hi_uk_news_education_2191422.stm.txt" |     2 +
 ..._news.bbc.co.uk_2_hi_africa_2909769.stm.txt" |     2 +
 ...news.bbc.co.uk_2_hi_uk_news_2509561.stm.txt" |     2 +
 .../txt/News/79NewsEnnajiM_Maghreb_EN.txt.txt   |     2 +
 .../News/80NewsGoalcom_MessiTop50_EN.txt.txt    |     2 +
 .../txt/News/81NewsGueyeA_BlackPete_EN.txt.txt  |     2 +
 .../News/82NewsLeM_OrbanGoldmanSachs_EN.txt.txt |     2 +
 ...sMendesFrancoJ_HaitiBeyondCapital_EN.txt.txt |     2 +
 ...NewsMillerH_FrankensteinTradition_EN.txt.txt |     2 +
 .../85NewsRabinovichI_IranNuclear_EN.txt.txt    |     2 +
 .../txt/News/86NewsRian_IranCutsOil_EN.txt.txt  |     2 +
 .../87NewsRian_MedvedevDismisses_EN.txt.txt     |     2 +
 .../News/88NewsWiki_KaradzicArrest_EN.txt.txt   |     2 +
 .../txt/Opac/89OpacStallman_FreeSoft_EN.txt.txt |     2 +
 .../txt/Opac/90OpacTeam_Berlin_EN.txt.txt       |     2 +
 .../txt/Opac/91OpacTeam_Budapest_EN.txt.txt     |     2 +
 .../Tedi/100TediOConnellA_Quantum_EN.txt.txt    |     2 +
 .../txt/Tedi/101TediRoslingH_Poverty_EN.txt.txt |     2 +
 .../102TediSinclairC_OpenArchitech_EN.txt.txt   |     2 +
 .../92TediAndersonEt_NuclearEnergy_EN.txt.txt   |     2 +
 .../Tedi/93TediBelcherA_Batteries_EN.txt.txt    |     2 +
 .../txt/Tedi/94TediGellMannM_Lang_EN.txt.txt    |     2 +
 .../Tedi/95TediJakubowskiM_OpenTech_EN.txt.txt  |     2 +
 .../Tedi/96TediJordanC_ShockStats_EN.txt.txt    |     2 +
 .../txt/Tedi/97TediMcKeanE_Dict_EN.txt.txt      |     2 +
 .../txt/Tedi/98TediMillerA_News_EN.txt.txt      |     2 +
 .../Tedi/99TediNatchweyJ_WarPhotos_EN.txt.txt   |     2 +
 .../txt/Tele/103TeleHTC_Manual_0_EN.txt.txt     |     2 +
 .../txt/Tele/104TeleHTC_Manual_10_EN.txt.txt    |     2 +
 .../txt/Tele/105TeleHTC_Manual_11_EN.txt.txt    |     2 +
 .../txt/Tele/106TeleHTC_Manual_12_EN.txt.txt    |     2 +
 .../txt/Tele/107TeleHTC_Manual_13_EN.txt.txt    |     2 +
 .../txt/Tele/108TeleHTC_Manual_1_EN.txt.txt     |     2 +
 .../txt/Tele/109TeleHTC_Manual_2_EN.txt.txt     |     2 +
 .../txt/Tele/110TeleHTC_Manual_3_EN.txt.txt     |     2 +
 .../txt/Tele/111TeleHTC_Manual_4_EN.txt.txt     |     2 +
 .../txt/Tele/112TeleHTC_Manual_5_EN.txt.txt     |     2 +
 .../txt/Tele/113TeleHTC_Manual_6_EN.txt.txt     |     2 +
 .../txt/Tele/114TeleHTC_Manual_7_EN.txt.txt     |     2 +
 .../txt/Tele/115TeleHTC_Manual_8_EN.txt.txt     |     2 +
 .../txt/Tele/116TeleHTC_Manual_9_EN.txt.txt     |     2 +
 .../txt/Teli/117TeliApple_iPhone4_EN.txt.txt    |     2 +
 .../txt/Tels/118TelsGoog_Answer_0545_EN.txt.txt |     2 +
 .../txt/Tels/119TelsGoog_Answer_0a46_EN.txt.txt |     2 +
 .../txt/Tels/120TelsGoog_Answer_0bcf_EN.txt.txt |     2 +
 .../txt/Tels/121TelsGoog_Answer_1851_EN.txt.txt |     2 +
 .../txt/Tels/122TelsGoog_Answer_1b37_EN.txt.txt |     2 +
 .../txt/Tels/123TelsGoog_Answer_2feb_EN.txt.txt |     2 +
 .../txt/Tels/124TelsGoog_Answer_3024_EN.txt.txt |     2 +
 .../txt/Tels/125TelsGoog_Answer_4b5b_EN.txt.txt |     2 +
 .../txt/Tels/126TelsGoog_Answer_596f_EN.txt.txt |     2 +
 .../txt/Tels/127TelsGoog_Answer_5b4c_EN.txt.txt |     2 +
 .../txt/Tels/128TelsGoog_Answer_70d7_EN.txt.txt |     2 +
 .../txt/Tels/129TelsGoog_Answer_770f_EN.txt.txt |     2 +
 .../txt/Tels/130TelsGoog_Answer_91e4_EN.txt.txt |     2 +
 .../txt/Tels/131TelsGoog_Answer_94d0_EN.txt.txt |     2 +
 .../txt/Tels/132TelsGoog_Answer_a67e_EN.txt.txt |     2 +
 .../133UnitGass_Resolution_62250_EN.txt.txt     |     2 +
 .../txt/Unit/134UnitUnat_Charter_EN.txt.txt     |     2 +
 .../txt/Unit/135UnitUnat_HumanRights_EN.txt.txt |     2 +
 .../txt/Unit/136UnitUnat_StatuteICOJ_EN.txt.txt |     2 +
 ...indParaschivoiuEt_DarrieusTurbine_EN.txt.txt |     2 +
 .../Wind/138WindPurohitEt_CDMIndia_EN.txt.txt   |     2 +
 ...ndRiadhEt_ContrarotatingConverter_EN.txt.txt |     2 +
 ...te\" url=\"http:__vue.org.uk_carlos.htm.txt" |     2 +
 ...ived_game.asp?MatchID=89&Season=2002_03.txt" |     2 +
 ...thoholidays.co.uk_ShowDetails.asp?id=96.txt" |     2 +
 ...news.org.uk_news_news_detail.asp?nid=22.txt" |     2 +
 ...url=\"http:__www.benhs.org.uk_anex.html.txt" |     2 +
 ..._guides_guides.php?subject=ratsthatbite.txt" |     2 +
 ...es_selected_cases_PCA_sc9903_c682b.html.txt" |     2 +
 ...l.cam.ac.uk_call_translation_toolkit_6_.txt" |     2 +
 ..._nnr-scotland_news_detail.asp?newsID=79.txt" |     2 +
 ...c.uk_Conferences_BPCSR05_submission.htm.txt" |     2 +
 ...co.uk_flatfiles_paulpearcetributes.aspx.txt" |     2 +
 ...resence.co.uk_phpBB2_viewtopic.php?t=97.txt" |     2 +
 ...loan.co.uk_article_Mortgages-1212.shtml.txt" |     2 +
 ...www.brainbashers.co.uk_droodlesprev.asp.txt" |     2 +
 ...onics.org.uk_newsletter_NoticeBoard.php.txt" |     2 +
 ..."http:__www.snh.org.uk_calendar_jul.asp.txt" |     2 +
 ..._Northamptonshire_jobs-in-Brackley.html.txt" |     2 +
 ...timatefilm_chart_details.php?ranking=65.txt" |     2 +
 ....co.uk_online_pr_online_pr.ehtml?o=1647.txt" |     2 +
 ...ling_human_organ_donation_dilemma_.html.txt" |     2 +
 ...www.sscs.bham.ac.uk_phsi_eating_bmi.htm.txt" |     2 +
 ...elected_cases_HSC_IC0107_pt1-e2242.html.txt" |     2 +
 ...org.uk_index.asp?contentid=21&menuid=21.txt" |     2 +
 ...d\" url=\"http:__www.ebe.org.uk_ccn.htm.txt" |     2 +
 ...humberland.gov.uk_vg_text_northpen.html.txt" |     2 +
 ..._reports_1992_theatre_museum_index.html.txt" |     2 +
 ...easynet.co.uk_jim.shead_River-Arun.html.txt" |     2 +
 ...wson-cruttenden.co.uk_conveyancing.html.txt" |     2 +
 ...radio.co.uk_acatalog_Vert_Arno_Ant.html.txt" |     2 +
 ...pendent.co.uk_europe_article1192096.ece.txt" |     2 +
 ..."http:__www.nsbapty.co.uk_Supp-Samp.htm.txt" |     2 +
 ...cp=.._swt_&cg=_&sim=&id=487&pagetype=27.txt" |     2 +
 ...p:__www.surf4wine.co.uk_Eben_Sadie.html.txt" |     2 +
 ...8%2D10+09%3A45%3A00&Logo=0&sort=cp&pg=1.txt" |     2 +
 ...codyfi.org.uk_commfirstactionplanpr.htm.txt" |     2 +
 ...rl=\"http:__www.sefton.gov.uk_page&3630.txt" |     2 +
 ...ective.org.uk_dwboard_messages_112.html.txt" |     2 +
 ...rsale-investment.org.uk_Plot-Sales.html.txt" |     2 +
 ...rdirectory.co.uk_car-leasing-jamjar.htm.txt" |     2 +
 ...\"http:__www.nta.nhs.uk_news_020624.htm.txt" |     2 +
 ....ac.uk_law_lawpages_Victim_Support.html.txt" |     2 +
 ...strialnetworking.co.uk_mag_v7-2_p7.html.txt" |     2 +
 ...php?location=_news_archive_20040628.htm.txt" |     2 +
 ...:__www.idler.co.uk_archives_?page_id=18.txt" |     2 +
 ...t.co.uk_support_index.pl?page=mailboxes.txt" |     2 +
 ...archives_2005_11_backstagebbccou_2.html.txt" |     2 +
 ...rk.co.uk_Health_default.asp?article=135.txt" |     2 +
 ...ingexperience.org.uk_learning_first.php.txt" |     2 +
 ...rtle.co.uk_edfest2006_terrysaunders.htm.txt" |     2 +
 ...s.gov.uk_legal_section21_chapter_f.html.txt" |     2 +
 ...www.tropicalfishcentre.co.uk_Plants.htm.txt" |     2 +
 ...org.uk_bba_jsp_polopoly.jsp?d=155&a=493.txt" |     2 +
 ...__www.burpham.surrey.sch.uk_potter.html.txt" |     2 +
 ...yId=14542&taggingType=4&contentId=11208.txt" |     2 +
 ...o.uk_catalogue_walker_walkerreviews.htm.txt" |     2 +
 ...reports_Spain_andalucia6_and-oct-03.htm.txt" |     2 +
 ...rvicesindex.co.uk_newsletter_aug04.html.txt" |     2 +
 ...x.cfm?fuseaction=details&nNewsID=560539.txt" |     2 +
 ...k_artists_artistspage.php?ID=204&page=3.txt" |     2 +
 ...re.co.uk_buying-property-continent.html.txt" |     2 +
 ....co.uk_usability_articles_print_wud.asp.txt" |     2 +
 ...i_3736cbd2e5895cf49854f8d70494bae7.html.txt" |     2 +
 ...am.pwp.blueyonder.co.uk_cx500_oil_pump_.txt" |     2 +
 ...index.php?name=News&file=article&sid=34.txt" |     2 +
 ....org.uk_scgn_articles_9902_inbrief.html.txt" |     2 +
 ...e.gov.uk_tmbc6_cycling_withoutmycar.htm.txt" |     2 +
 ...ersonal_pages_Ifan1_Booth_Notebooks.htm.txt" |     2 +
 ...le_3603_reduce-cancer-reoccurrence.html.txt" |     2 +
 ...discovery.co.uk_job.aspx?jid=11535&cd=1.txt" |     2 +
 ...www.syscom.plc.uk_solutions_distrib.asp.txt" |     2 +
 ...ay.jsp?section=Banking&article_id=64923.txt" |     2 +
 ...olutec.co.uk_06_chairman.asp?thesub=6.0.txt" |     2 +
 ....truststfc.co.uk_meeting_27_09_2006.php.txt" |     2 +
 ...ition_personal_experiences_michael.html.txt" |     2 +
 ..._library_history_library_history_3.html.txt" |     2 +
 ...listunitynetwork.co.uk_news_g8jepps.htm.txt" |     2 +
 ...:__www.weirdwiltshire.co.uk_250703.html.txt" |     2 +
 ...idwifery_lsa-guidelines_maternal-deaths.txt" |     2 +
 ...+Ltd&frmBPE=&frmCD=N&mopt=dpe&dpid=2302.txt" |     2 +
 ..." url=\"http:__www.siba.co.uk_about.asp.txt" |     2 +
 ...ave_Nirvana_2_Mizuno_Running_Shoes.html.txt" |     2 +
 ...earch.org.uk_?lid=1944&tmpl=ddmainprint.txt" |     2 +
 ...departments_services_campusenvironment_.txt" |     2 +
 ...etails.asp?ED=Arts+and+Crafts&offset=66.txt" |     2 +
 ...t_DBID_17ea4c66d7bd2c0aeb4513c89cb01afd.txt" |     2 +
 ....fst.rdg.ac.uk_news-archive-2004-11.htm.txt" |     2 +
 ..._www.aslib.co.uk_training_careers_9.htm.txt" |     2 +
 ...profile_Can-I-spend-time-on-an-elective.txt" |     2 +
 ...comesdata.co.uk_europe_duediligence.htm.txt" |     2 +
 ...nnine.demon.co.uk_NPC_1982_MEXICOSP.HTM.txt" |     2 +
 ...p:__www.hsl.gov.uk_publications_car.htm.txt" |     2 +
 ...w.baronage.co.uk_bphtm-01_const-02.html.txt" |     2 +
 ...14C83960FBA06562FFBA3B67013B5558FE96AD7.txt" |     2 +
 ...myleedsjobs.co.uk_jobdetails-11834.html.txt" |     2 +
 ...p:__www.paperairplanes.co.uk_orplan.php.txt" |     2 +
 ...\"http:__www.eca.ac.uk_tacitus_news.htm.txt" |     2 +
 ...casino-avenue.co.uk_2004_06_duuuuh.html.txt" |     2 +
 ...uk_lifeevent_penret_penreform_5_reg.asp.txt" |     2 +
 ...earch.co.uk_projects_T5_excavation.html.txt" |     2 +
 ...ww.redcross.org.uk_section.asp?id=49633.txt" |     2 +
 ...orum.org.uk_jobs_forestsmonitor2001.htm.txt" |     2 +
 ...um.org.uk_kids_detail.asp?ContentID=189.txt" |     2 +
 ...hetelly.co.uk_interviews_markwright.htm.txt" |     2 +
 ...__www.sitcom.co.uk_tlc_characters.shtml.txt" |     2 +
 ...ampian.co.uk_whiskycountry_ess_walk.htm.txt" |     2 +
 ...arty.org.uk_sgpnewsarticle20051222a.htm.txt" |     2 +
 ...ployment-solicitors.co.uk_Employer1.htm.txt" |     2 +
 ....uk_news_Food_Sounds_So_Good_at_DArcys_.txt" |     2 +
 ...ttp:__www.sweetsforu.co.uk_shipping.php.txt" |     2 +
 ...rg.uk_Listings.aspx?index=387&item=2929.txt" |     2 +
 ....uk_product.aspx?catno=53&prod=HCAA6241.txt" |     2 +
 ....uk_main_en_att-provider-ROMA_6913.html.txt" |     2 +
 ...rt-works.org.uk_artworks_z030703b.shtml.txt" |     2 +
 ...lier_booth_miscellaneous_wristgang.html.txt" |     2 +
 ...w.changingdiabetes.co.uk_view.asp?ID=92.txt" |     2 +
 ...ets.co.uk_las%20new%20bionix%20page.htm.txt" |     2 +
 ...=37&sql=&sortup=sorttitle&bookstatus=OK.txt" |     2 +
 ...org.uk_cheshire_proj_harvest_survey.htm.txt" |     2 +
 ...emydown.co.uk_clancomments.php?id=35113.txt" |     2 +
 ...p:__www.scis.org.uk_search_menu_new.asp.txt" |     2 +
 ...ckbankmembership_clickbankprotector.htm.txt" |     2 +
 ...lewa.co.uk_project5_teachers_T5-0-1.htm.txt" |     2 +
 ...dex.php?category=campaigns&c=i&uid=2130.txt" |     2 +
 ...ww.kent-ccc.co.uk_news_story.php?id=660.txt" |     2 +
 ...t.asp?WCI=SiteHome&ID=9908&PageID=56638.txt" |     2 +
 ...k_jsp_id_0340894342_Divine_Madness.html.txt" |     2 +
 ...:__www.newble.co.uk_chalmers_innes.html.txt" |     2 +
 ....searchenginespy.co.uk_article0027.html.txt" |     2 +
 ...pus_search_document.php?documentid=1211.txt" |     2 +
 ...lyheritage_forum_topic.asp?TOPIC_ID=26&.txt" |     2 +
 ...andhousesforsalelimousinabn0509263.html.txt" |     2 +
 ...cedevelopment_managerialfacilities.aspx.txt" |     2 +
 ...fety_various_rass_kmweb_safety_msds.htm.txt" |     2 +
 ...in.co.uk_index.php?id=23&L=3&article=13.txt" |     2 +
 ...rAreaNo=1053,2011&strKeyword=PS2006_4_3.txt" |     2 +
 ...tp:__www.itreviews.co.uk_games_g232.htm.txt" |     2 +
 ...co.uk_2002_09_to-quote-the-four-seasons.txt" |     2 +
 ...lt.aspx?group_id=16538&article_id=21979.txt" |     2 +
 ...arch_staff_jpf_papers_paper26_index.php.txt" |     2 +
 ..._www.all-energy.co.uk_newsletter45.html.txt" |     2 +
 ...rker.org.uk_article.php?article_id=8138.txt" |     2 +
 ....setdanceteacher.co.uk_newmarketmez.htm.txt" |     2 +
 ..._interpro_DisplayIproEntry?ac=IPR002824.txt" |     2 +
 ...www.garthyfog.co.uk_mawddach_valley.htm.txt" |     2 +
 ...aving_expeditions_jura05_jura_circ1.php.txt" |     2 +
 ...a-online.co.uk_viewnews.cfm?news_id=177.txt" |     2 +
 ...omPage.aspx?PageID=24163&sectionID=4585.txt" |     2 +
 ...ews_news_search.php?search=&start=12080.txt" |     2 +
 ...linc4info.org.uk_cms_pages_sitemap.html.txt" |     2 +
 ....ccp4.ac.uk_courses_IUCr2005_index.html.txt" |     2 +
 ...ve-engineering.co.uk_html_training.html.txt" |     2 +
 ..." url=\"http:__www.lanpac.co.uk_csi.php.txt" |     2 +
 ...org.uk_html_files_501_project_info.html.txt" |     2 +
 ...s.co.uk_showpage.asp?showdocumentid=196.txt" |     2 +
 ....ac.uk_star_docs_sun232.htx_node17.html.txt" |     2 +
 ..._cmhansrd_vo000405_debtext_00405-07.htm.txt" |     2 +
 ...RESBYTERY%20OF%20PENPONT%20p.%20672.htm.txt" |     2 +
 ...w.wessingtoncryogenics.co.uk_serv01.htm.txt" |     2 +
 ..._content_index.jsp?contentid=1999276669.txt" |     2 +
 ...tratford-upon-avon.co.uk_static_481.htm.txt" |     2 +
 ...group.co.uk_office-support-recruitment_.txt" |     2 +
 ...talog_product_info.php?products_id=2981.txt" |     2 +
 ..._fourwheels_formula1_article.asp?a=1327.txt" |     2 +
 ...id_docs_api_javax_swing_JSplitPane.html.txt" |     2 +
 ...online.co.uk_60_66_67_articles_7335.php.txt" |     2 +
 ...ww.iae.co.uk_news_designedforthejob.htm.txt" |     2 +
 ...uk_Case_Studies_studies_Ford3_Ford3.asp.txt" |     2 +
 ...ffee.co.uk_product.php?xProd=21&xSec=22.txt" |     2 +
 ...usability_aboutus_usability_aboutus.asp.txt" |     2 +
 ...d_story.asp?latestchapter=12&subarea=11.txt" |     2 +
 ...tt.co.uk_pages_searchdetails.asp?ID=776.txt" |     2 +
 ...forum.org.uk_userpage1.cfm?item_id=1913.txt" |     2 +
 ...choolhouse.org.uk_law_not_enrolled.html.txt" |     2 +
 ...lendar_archive_article.cfm?articleId=52.txt" |     2 +
 ...tp:__www.lathes.co.uk_beaver_page5.html.txt" |     2 +
 ...ww.hasslefreeminiatures.co.uk_rules.php.txt" |     2 +
 ...k_modules_event_viewevent.php?eveid=109.txt" |     2 +
 ...-cheats_Gamecube-(hardware)-Cheats.html.txt" |     2 +
 ...www.omega.co.uk_ppt_pptsc.asp?ref=LE902.txt" |     2 +
 ...rum_poster.cfm?sort=creatasc&poster=101.txt" |     2 +
 ....uk_directory_prof_issues_blreview.html.txt" |     2 +
 ...d_to_back_london's_olympic_2012_bid.htm.txt" |     2 +
 ...rum=106&thread=9757638&message=11724737.txt" |     2 +
 ...51_20553_100_10012_10010_category_10010.txt" |     2 +
 ....co.uk_news_industry_2005_ind505016.htm.txt" |     2 +
 ...uea.ac.uk_eas_events_litfestspr04.shtml.txt" |     2 +
 ...imberry.co.uk_Dotnetlectures_Index.aspx.txt" |     2 +
 ....lathes.co.uk_wolfjahnmiller_page2.html.txt" |     2 +
 ...te_cms_newsarticleview.asp?article=2173.txt" |     2 +
 ...ttp:__www.anweb.co.uk_l_04_c3_c3a10.htm.txt" |     2 +
 ...subject_modernlanguages_course_s1s2.htm.txt" |     2 +
 ...p:__www.law.warwick.ac.uk_ltj_4-1m.html.txt" |     2 +
 ...shipfoundation.org.uk_main_news.php?n20.txt" |     2 +
 ...ttp:__www.arnside-online.co.uk_care.htm.txt" |     2 +
 ...ort.co.uk_products_leatherman_micra.php.txt" |     2 +
 ...g.uk_columnists_docdiary2.php?docId=103.txt" |     2 +
 ...ne.co.uk_article_articleview_1733_1_153.txt" |     2 +
 ...-survey.pwp.blueyonder.co.uk_P_mugo.htm.txt" |     2 +
 ....free-internet.co.uk_email_sendmail.htm.txt" |     2 +
 ...esources_ums_PythonDoc_api_threads.html.txt" |     2 +
 ...l?in_article_id=405873&in_page_id=50002.txt" |     2 +
 ...rvices.gcal.ac.uk_synergy_03_scwbl.html.txt" |     2 +
 ..._BF_NEWSART_view.asp?Q=BF_NEWSART_95582.txt" |     2 +
 ...ware.co.uk_printpage.asp?REF=_group.asp.txt" |     2 +
 ...__www.dba.org.uk_aboutdba_chriswood.asp.txt" |     2 +
 ...ulture_books_0804_110804_food_books.htm.txt" |     2 +
 ..._www.newtsnni.gov.uk_actionplan_04b.htm.txt" |     2 +
 ..._KS3_databases_relational_databases.htm.txt" |     2 +
 ...on_pages_SchemesofWork_KS4_skillswl.htm.txt" |     2 +
 ...en_unusual-gadgets_mood-light-tile.html.txt" |     2 +
 ...cle&sid=175&mode=thread&order=0&thold=0.txt" |     2 +
 ..."http:__www.engender.org.uk_justice.htm.txt" |     2 +
 ...:__www.heros.org.uk_home_sub.asp?page=2.txt" |     2 +
 ...rg.uk_clinical_prac_mar_05_mar05_08.htm.txt" |     2 +
 ...ww.cdp.bham.ac.uk_About_CDP_methods.htm.txt" |     2 +
 ...k_caminfo_blueprint_articles.asp?ID=807.txt" |     2 +
 ...islation_scotland_acts2002_20017--b.htm.txt" |     2 +
 ...ures.co.uk_dog-pictures_shiba_inu.shtml.txt" |     2 +
 ...wbale-building.co.uk_index.php?page=faq.txt" |     2 +
 ...co.uk_londonunderuk_tfl_our_careers.asp.txt" |     2 +
 ...www.environment.bham.ac.uk_extindex.htm.txt" |     2 +
 ...bitions_Aug%2006%2004_James%20Cauty.htm.txt" |     2 +
 ...__www.sincuser.f9.co.uk_050_lastwrd.htm.txt" |     2 +
 ...__www.ocdaction.org.uk_skin-picking.htm.txt" |     2 +
 ..."http:__www.greenparty.org.uk_news_2033.txt" |     2 +
 ...ed.ac.uk_linguist_issues_17_17-229.html.txt" |     2 +
 ...s_aRelease.asp?akey=2026&Mon=01_07_2004.txt" |     2 +
 ...rl=\"http:__www.lpt.nhs.uk_service5.php.txt" |     2 +
 ....uk_resources_reportdetails.asp?id=1039.txt" |     2 +
 ...ctech.co.uk_siemens_hosted_exchange.htm.txt" |     2 +
 ...k_archives_2006_01_chomsky_intervi.html.txt" |     2 +
 .../0syndicate_bmw-s-and-chinese-justice.txt    |     2 +
 .../synd/10syndicate_france-s-fourth-moment.txt |     2 +
 ...ong-idea-of-france-by-brigitte-granville.txt |     2 +
 ...2syndicate_hollywood-s-favorite-villains.txt |     2 +
 ...te_japanese-lessons-for-china-s-currency.txt |     2 +
 ...rence-from-cuba-to-iran-by-joseph-s--nye.txt |     2 +
 ...yndicate_merkel-in-china-by-sanjaya-baru.txt |     2 +
 ...16syndicate_ms--europe-or-frau-germania-.txt |     2 +
 ...pply-chain-by-andrew-sheng-and-geng-xiao.txt |     2 +
 ...te_russia-s-ukrainian-path-to-the-future.txt |     2 +
 ...-europe-regulate-sovereign-wealth-funds-.txt |     2 +
 ...ndicate_china-and-a-new-balance-of-power.txt |     2 +
 ...ate_the--browning--of-african-technology.txt |     2 +
 .../txt/synd/21syndicate_the-china-question.txt |     2 +
 ...icate_the-end-of-the-russia-china-debate.txt |     2 +
 ...cate_the-making-of-china-s-trade-deficit.txt |     2 +
 ...e_the-return-of-franco-german-leadership.txt |     2 +
 .../25syndicate_the-transatlantic-index.txt     |     2 +
 .../26syndicate_will-russia-save-the-west-.txt  |     2 +
 ...ina-and-russia-in-the-new-world-disorder.txt |     2 +
 ...japan-protests-islands-by-liah-greenfeld.txt |     2 +
 .../synd/4syndicate_china-s-cyber-warriors.txt  |     2 +
 .../synd/5syndicate_china-s-gift-to-europe.txt  |     2 +
 .../6syndicate_china-s-threat-to-russia.txt     |     2 +
 ...s-vital-french-connection-by-kemal-dervi.txt |     2 +
 .../synd/8syndicate_exchange-rate-disorder.txt  |     2 +
 ...cate_france-and-germany-must-act-in-iraq.txt |     2 +
 ...ted_a_j_jacobs_year_of_living_biblically.txt |     2 +
 ...n_kay_shares_a_powerful_idea_about_ideas.txt |     2 +
 ...d_alex_tabarrok_foresees_economic_growth.txt |     2 +
 .../txt/ted/143ted_amy_tan_on_creativity.txt    |     2 +
 ..._barry_schwartz_on_the_paradox_of_choice.txt |     2 +
 ...njamin_wallace_on_the_price_of_happiness.txt |     2 +
 .../146ted_bono_s_call_to_action_for_africa.txt |     2 +
 ...147ted_dan_dennett_cute_sexy_sweet_funny.txt |     2 +
 ...ents_a_better_way_to_harvest_bone_marrow.txt |     2 +
 ...es_his_ted_prize_wish_once_upon_a_school.txt |     2 +
 .../150ted_don_norman_on_design_and_emotion.txt |     2 +
 ...goodwin_on_learning_from_past_presidents.txt |     2 +
 ...elliot_krane_the_mystery_of_chronic_pain.txt |     2 +
 ..._zeisel_on_the_playful_search_for_beauty.txt |     2 +
 ...x_moonie_diane_benscoter_how_cults_think.txt |     2 +
 .../txt/ted/155ted_hans_rosling_at_state.txt    |     2 +
 ..._helen_fisher_tells_us_why_we_love_cheat.txt |     2 +
 ...ed_isabel_allende_tells_tales_of_passion.txt |     2 +
 ...8ted_jessi_arrington_wearing_nothing_new.txt |     2 +
 ...nderkoffler_drive_3d_data_with_a_gesture.txt |     2 +
 ...joseph_lekuton_tells_a_parable_for_kenya.txt |     2 +
 .../ted/161ted_julia_sweeney_has_the_talk.txt   |     2 +
 ...en_robinson_says_schools_kill_creativity.txt |     2 +
 .../txt/ted/163ted_marcin_jakubowski.txt        |     2 +
 ...h_10_things_you_didn_t_know_about_orgasm.txt |     2 +
 ...sandel_the_lost_art_of_democratic_debate.txt |     2 +
 ...hael_shermer_on_believing_strange_things.txt |     2 +
 ...lenny_investigates_global_crime_networks.txt |     2 +
 ...at_hallucination_reveals_about_our_minds.txt |     2 +
 ...cca_saxe_how_brains_make_moral_judgments.txt |     2 +
 .../ted/170ted_robert_thurman_on_compassion.txt |     2 +
 ...y_sutherland_life_lessons_from_an_ad_man.txt |     2 +
 ...d_sean_gourley_on_the_mathematics_of_war.txt |     2 +
 ...s_how_he_found_the_true_face_of_leonardo.txt |     2 +
 ...wolfram_computing_a_theory_of_everything.txt |     2 +
 .../txt/ted/175ted_tom_wujec_build_a_tower.txt  |     2 +
 ...ujec_on_3_ways_the_brain_creates_meaning.txt |     2 +
 .../txt/ted/177ted_william_li.txt               |     2 +
 ...pany-wizard.co.uk_SEO_Link-Exchange.htm.txt" |     2 +
 ...org.uk_newspub_story.cfm?id=353&sid=123.txt" |     2 +
 ...ngconcern.org.uk_news_news_scots06.html.txt" |     2 +
 ..._in_the_Community_Joan_Wolstenholme.htm.txt" |     2 +
 ...t.asp?WCI=SiteHome&ID=4337&PageID=21591.txt" |     2 +
 ...ha.freeserve.co.uk_glh_233_mckenna.html.txt" |     2 +
 .../txt/un/A/178un_A_AC252_L13.txt              |     2 +
 .../txt/un/A/179un_A_C3_61_L17.txt              |     2 +
 .../txt/un/A/180un_APLC_MSP2_2000_1.txt         |     2 +
 .../txt/un/C/181un_CCW_APII_CONF4_SR2.txt       |     2 +
 .../style_recognizer/txt/un/C/182un_CD_1605.txt |     2 +
 .../style_recognizer/txt/un/C/183un_CD_1606.txt |     2 +
 .../style_recognizer/txt/un/C/184un_CD_1672.txt |     2 +
 .../txt/un/C/185un_CD_PV1080.txt                |     2 +
 .../txt/un/C/186un_CD_PV1157.txt                |     2 +
 .../txt/un/C/187un_CD_PV857.txt                 |     2 +
 .../txt/un/C/188un_CD_PV861.txt                 |     2 +
 .../txt/un/C/189un_CEDAW_C_2002_II_3_ADD2.txt   |     2 +
 .../txt/un/C/190un_CES_SEM52_2.txt              |     2 +
 .../txt/un/C/191un_CES_SEM52_3.txt              |     2 +
 .../style_recognizer/txt/un/C/192un_CLCS_34.txt |     2 +
 .../txt/un/C/193un_CRC_GC_2003_4.txt            |     2 +
 .../txt/un/D/194un_DP_2002_34.txt               |     2 +
 .../txt/un/E/195un_E_C16_2002_2.txt             |     2 +
 .../txt/un/E/196un_E_CN17_2002_PC2_19.txt       |     2 +
 .../txt/un/E/197un_E_CN3_2003_25.txt            |     2 +
 .../txt/un/E/198un_E_CN4_2003_G_12.txt          |     2 +
 .../txt/un/E/199un_E_CN4_2004_G_21.txt          |     2 +
 .../txt/un/F/200un_FCCC_SBI_2002_15.txt         |     2 +
 .../txt/un/F/201un_FCCC_SBI_2002_9.txt          |     2 +
 .../txt/un/F/202un_FCCC_SBI_2004_L27.txt        |     2 +
 .../txt/un/H/203un_HRI_CORE_1_ADD117.txt        |     2 +
 .../txt/un/H/204un_HRI_CORE_1_ADD123.txt        |     2 +
 .../txt/un/H/205un_HSP_GC_19_2_ADD2.txt         |     2 +
 .../style_recognizer/txt/un/I/206un_IDB25_6.txt |     2 +
 .../txt/un/I/207un_IDB35_13.txt                 |     2 +
 .../txt/un/I/208un_ISBA_8_A_1.txt               |     2 +
 .../txt/un/N/209un_NPT_CONF2005_PCI_10.txt      |     2 +
 .../txt/un/P/210un_PBC24_1_ADD1.txt             |     2 +
 .../txt/un/P/211un_PBC24_4-IDB35_4.txt          |     2 +
 .../txt/un/P/212un_PBC_2_SLE_9.txt              |     2 +
 .../txt/un/P/213un_PCNICC_2000_1_ADD1.txt       |     2 +
 .../txt/un/P/214un_PCNICC_2001_WGAPIC_L1.txt    |     2 +
 .../txt/un/S/215un_S_2000_475.txt               |     2 +
 .../txt/un/S/216un_S_2000_817.txt               |     2 +
 .../txt/un/S/217un_S_2001_1038-A_56_521.txt     |     2 +
 .../txt/un/S/218un_S_2001_1039.txt              |     2 +
 .../txt/un/S/219un_S_2001_1317.txt              |     2 +
 .../txt/un/S/220un_S_2001_70-A_55_740.txt       |     2 +
 .../txt/un/S/221un_S_2001_947.txt               |     2 +
 .../txt/un/S/222un_S_2002_1141.txt              |     2 +
 .../txt/un/S/223un_S_2002_1180.txt              |     2 +
 .../txt/un/S/224un_S_2002_238-A_56_858.txt      |     2 +
 .../txt/un/S/225un_S_2002_313-A_56_888.txt      |     2 +
 .../txt/un/S/226un_S_2002_573-A_56_960.txt      |     2 +
 .../txt/un/S/227un_S_2003_326-A_ES-10_222.txt   |     2 +
 .../txt/un/S/228un_S_2006_490.txt               |     2 +
 .../txt/un/S/229un_S_2009_208-A_63_820.txt      |     2 +
 .../txt/un/S/230un_SAICM_ICCM2_INF_18.txt       |     2 +
 .../txt/un/S/231un_S_PV4596.txt                 |     2 +
 .../txt/un/S/232un_S_PV4646.txt                 |     2 +
 .../txt/un/S/233un_S_PV4684RESUMPTION1.txt      |     2 +
 .../txt/un/S/234un_S_PV4736.txt                 |     2 +
 .../txt/un/S/235un_S_PV4903.txt                 |     2 +
 .../txt/un/S/236un_S_PV4944.txt                 |     2 +
 .../txt/un/S/237un_S_PV5627.txt                 |     2 +
 .../txt/un/S/238un_ST_SGB_2002_9.txt            |     2 +
 .../txt/un/T/239un_TD_B_49_11.txt               |     2 +
 .../txt/un/T/240un_TD_B_49_SC2_L1.txt           |     2 +
 .../txt/un/T/241un_TD_B_COM3_54.txt             |     2 +
 .../txt/un/T/242un_TD_B_EX28_L1.txt             |     2 +
 .../txt/un/T/243un_TESTGVADUTY001.txt           |     2 +
 .../txt/un/U/244un_UNEP_GC_22_8_ADD2.txt        |     2 +
 .../txt/un/U/245un_UNEP_GC_24_2.txt             |     2 +
 .../txt/un/U/246un_UNEP_POPS_INC6_22.txt        |     2 +
 ...r.ac.uk_sppa_sppa_responsibilities.html.txt" |     2 +
 ...tomer_terms_index.omp?cid=1146151223400.txt" |     2 +
 ...Cmach_Backissues_j005_Articles_hall.htm.txt" |     2 +
 ...k_archives_2006_06_met_chief_could.html.txt" |     2 +
 ...:__www.fenews.co.uk_newsview.asp?n=1014.txt" |     2 +
 ..._north_east_petscorner_news_140106.html.txt" |     2 +
 ...tp:__www.nursingbra.co.uk_emma_jane.htm.txt" |     2 +
 ..._care_worcestershire_easemore_road.html.txt" |     2 +
 ...k_comedy_comedians_comedian.aspx?id=429.txt" |     2 +
 ...ttp:__www.seequality.org.uk_gender.html.txt" |     2 +
 .../src/test/resources/taxonomies/fidelity.txt  |   503 +
 .../resources/taxonomies/musicTaxonomyRoot.csv  |     1 +
 .../tree_kernel/action-plan-doc-list.csv        |   119 +
 .../src/test/resources/tree_kernel/model.txt    | 10143 ++++++++++
 .../test/resources/tree_kernel/svm.run.mac.zip  |   Bin 0 -> 90804 bytes
 .../src/test/resources/tree_kernel/svm_classify |   Bin 0 -> 257356 bytes
 .../test/resources/tree_kernel/svm_classify.exe |   Bin 0 -> 136343 bytes
 .../resources/tree_kernel/svm_classify.linux    |   Bin 0 -> 275767 bytes
 .../src/test/resources/tree_kernel/svm_learn    |   Bin 0 -> 336068 bytes
 .../test/resources/tree_kernel/svm_learn.exe    |   Bin 0 -> 147778 bytes
 .../src/test/resources/tree_kernel/training.txt | 16350 +++++++++++++++++
 .../tree_kernel/training_pos_neg_sentiment.txt  | 15930 ++++++++++++++++
 615 files changed, 61149 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/README.md
----------------------------------------------------------------------
diff --git a/opennlp-similarity/README.md b/opennlp-similarity/README.md
new file mode 100644
index 0000000..65ccfdd
--- /dev/null
+++ b/opennlp-similarity/README.md
@@ -0,0 +1,157 @@
+# OpenNLP.Similarity Component
+
+It is a project under Apache OpenNLP which subjects results of parsing, part-of-speech tagging and rhetoric parsing to machine learning.
+It is leveraged in search, content generation & enrichment, chat bots and other text processing domains where relevance assessment task is a key.
+
+## What is OpenNLP.Similarity?
+
+OpenNLP.Similarity is an NLP engine which solves a number of text processing and search tasks based on OpenNLP and Stanford NLP parsers. It is designed to be used by a non-linguist software engineer to build linguistically-enabled: 
+<ul>
+<li>search engines</li>
+<li>recommendation systems</li>
+<li>dialogue systems</li>
+<li>text analysis and semantic processing engines</li>
+<li>data-loss prevention system</li>
+<li>content & document generation tools</li>
+<li>text writing style, authenticity, sentiment, sensitivity to sharing recognizers</li>
+<li>general-purpose deterministic inductive learner equipped with abductive, deductive and analogical reasoning which also embraces concept learning and tree kernel learning. </li>
+</ul>
+
+OpenNLP similarity provides a series of techniques to support the overall content pipeline, from text collection to cleaning, classification, personalization and distribution. Technology and implementation of content pipeline developed at eBay is described [here](https://github.com/bgalitsky/relevance-based-on-parse-trees/tree/master/examples/ContentPipeline.pdf). 
+## Installation
+ 0) Do [`git clone`](https://github.com/bgalitsky/relevance-based-on-parse-trees.git) to setup the environment including resources. Besides what you get from git, `/resources` directory requires some additional work:
+ 
+ 1) Download the main [jar](https://github.com/bgalitsky/relevance-based-on-parse-trees/blob/master/opennlp-similarity.11.jar).
+ 
+ 2) Set all necessary jars in /lib folder. Larger size jars are not on git so please download them from [Stanford NLP site](http://nlp.stanford.edu/)
+ <li>edu.mit.jverbnet-1.2.0.jar</li>
+ <li>ejml-0.23.jar</li>
+ <li>joda-time.jar</li>
+ <li>jollyday.jar</li>
+ <li>stanford-corenlp-3.5.2-models.jar</li>
+ <li>xom.jar</li>
+ The rest of jars are available via maven.
+ 
+ 3) Set up src/test/resources directory
+ - new_vn.zip needs to be unzipped
+ - OpenNLP models need to be downloaded into the directory 'models' from [here](http://opennlp.sourceforge.net/models-1.5/)
+  
+  As a result the following folders should be in in /resources:
+  As obtained [from git](https://github.com/bgalitsky/relevance-based-on-parse-trees/tree/master/src/test/resources):
+ <li>/new_vn (VerbNet)</li>
+ <li>/maps (some lookup files such as products, brands, first names etc.)</li>
+ <li>/external_rst (examples of import of rhetoric parses from other systems)</li>
+ <li>/fca (Formal Concept Analysis learning)</li>
+ <li>/taxonomies (for search support, taxonomies are auto-mined from the web)</li>
+ <li>/tree_kernel (for tree kernel learning, representation of parse trees, thickets and trained models)</li>
+  Manual downloading is also required for:
+  <li>/new_vn</li>
+  <li>/w2v (where word2vector model needs to be downloaded, if desired)</li>
+  
+ 4) Try running tests which will give you a hint on how to integrate OpenNLP.Similarity functionality into your application. You can start with [Matcher test](https://github.com/bgalitsky/relevance-based-on-parse-trees/blob/949bac8c2a41c21a1e54fec075f2966d693114a4/src/test/java/opennlp/tools/parse_thicket/matching/PTMatcherTest.java) and observe how long paragraphs can be linguistically matched (you can compare this with just an intersection of keywords)
+  
+ 5) Look at [example POMs](https://github.com/bgalitsky/relevance-based-on-parse-trees/tree/master/examples) for how to better integrate into your existing project
+  
+## Creating a simple project
+
+  Create a project from [MyMatcher.java](https://github.com/bgalitsky/relevance-based-on-parse-trees/tree/master/examples/MyMatcher.java).  
+ 
+## Engines and Systems of OpenNLP.Similarity
+
+### Main relevance assessment function
+It takes two texts and returns the cardinality of a maximum common subgraph representations of these texts. This measure is supposed to be much more accurate than keyword statistics, compositional semantic models word2vec because linguistic structure is taken into account, not just co-occurrences of keywords. 
+  [Matching class](https://github.com/bgalitsky/relevance-based-on-parse-trees/blob/master/src/main/java/opennlp/tools/parse_thicket/matching/Matcher.java) in [matching package] (https://github.com/bgalitsky/relevance-based-on-parse-trees/tree/master/src/main/java/opennlp/tools/parse_thicket/matching) has 
+
+`List<List<ParseTreeChunk>> assessRelevance(String para1, String para2)`
+
+function which returns the list of [common phrases between these paragraph]s.
+
+To avoid re-parsing the same strings and improve the speed, use
+
+`List<List<ParseTreeChunk>> assessRelevanceCache(String para1, String para2)`
+
+It operates on the level of sentences (giving [maximal common subtree](https://github.com/bgalitsky/relevance-based-on-parse-trees/tree/master/examples/Inferring_sem_prop_of_sentences.pdf)) and paragraphs (giving maximal common [sub-parse thicket](https://en.wikipedia.org/wiki/Parse_Thicket)). Maximal common sub-parse thicket is also represented as a [list of common phrases](https://github.com/bgalitsky/relevance-based-on-parse-trees/tree/master/examples/MachineLearningSyntParseTreesGalitsky.pdf).
+
+<li>Search results re-ranker based on linguistic similarity</li>
+<li>Request Handler for SOLR which used parse tree similarity</li>
+
+### Search engine
+The following set of functionalities is available to enable search with linguistic features. It is desirable when query is long (more than 4 keywords), logically complex, ambiguous or 
+<li>Search results re-ranker based on linguistic similarity</li>
+<li>Request Handler for SOLR which used parse tree similarity</li>
+<li>Taxonomy builder via learning from the web</li>
+<li>Appropriate rhetoric map of an answer verifier. If parts of the answer are located in distinct discourse units, this answer might be irrelevant even if all keywords are mapped</li>
+<li>Tree kernel learning re-ranker to improve search relevance within a given domain with pre-trained model</li>
+
+SOLR request handlers are available [here](https://github.com/bgalitsky/relevance-based-on-parse-trees/tree/master/src/main/java/opennlp/tools/similarity/apps/solr)
+
+Taxonomy builder is [here](https://github.com/bgalitsky/relevance-based-on-parse-trees/tree/master/src/main/java/opennlp/tools/similarity/apps/taxo_builder).
+ Examples of pre-built taxonomy are available in [this directory](https://github.com/bgalitsky/relevance-based-on-parse-trees/tree/master/src/test/resources/taxonomies). Please pay attention at taxonomies built for languages other than English. A [music taxonomy](https://github.com/bgalitsky/relevance-based-on-parse-trees/blob/master/src/test/resources/taxonomies/musicTaxonomyRoot.csv) is an example of the seed data for taxonomy building, and [this taxonomy hashmap dump](https://github.com/bgalitsky/relevance-based-on-parse-trees/tree/master/src/main/java/opennlp/tools/similarity/apps/taxo_builder/taxonomy.txt) is a good example of what can be automatically constructed. A paper on taxonomy learning is [here](https://github.com/bgalitsky/relevance-based-on-parse-trees/blob/master/examples/taxonomyBuilder.pdf). 
+ 
+#### Search results re-ranker
+Re-ranking scores similarity between a given `orderedListOfAnswers` and  `question`
+
+  `List<Pair<String,Double>> pairList = new ArrayList<Pair<String,Double>>();`
+  
+  `for (String ans: orderedListOfAnswers) {`
+  
+            `List<List<ParseTreeChunk>> similarityResult = m.assessRelevanceCache(question, ans);`
+            
+            `double score = parseTreeChunkListScorer.getParseTreeChunkListScoreAggregPhraseType(similarityResult);`
+            
+            `Pair<String,Double> p = new Pair<String, Double>(ans, score);`
+            
+            `pairList.add(p);`
+            
+        `}`
+        
+   `Collections.sort(pairList, Comparator.comparing(p -> p.getSecond()));`
+   
+   Then `pairList` is then ranked according to the linguistic relevance score. This score can be combined with other sources such as popularity, geo-proximity and others.
+
+### Content generator
+ It takes a topic, builds a taxonomy for it and forms a table of content. It then  mines the web for documents for each table of content item, finds relevant sentences and paragraphs and merges them into a document [package](https://github.com/bgalitsky/relevance-based-on-parse-trees/tree/master/src/main/java/opennlp/tools/similarity/apps). The resultant document has a TOC, sections, figures & captions and also a reference section. We attempt to reproduce how humans cut-and-paste content from the web while writing on a topic. 
+  Content generation has a [demo](http://37.46.135.20/)  and to run it from IDE start [here](https://github.com/bgalitsky/relevance-based-on-parse-trees/blob/master/src/main/java/opennlp/tools/similarity/apps/ContentGeneratorRunner.java). Examples of written documents are [here](http://37.46.135.20/wrt_latest/).
+  Another content generation option is about opinion data. Reviews are mined for, cross-bred and made "original" for search engines. This and general content generation is done for SEO purposes. [Review builder](https://github.com/bgalitsky/relevance-based-on-parse-trees/tree/master/src/main/java/opennlp/tools/apps/review_builder/ReviewBuilderRunner.java) composes fake reviews which are in turn should be recognized by a Fake Review detector
+
+### Text classifier / feature detector in text
+The [classifier code](https://github.com/bgalitsky/relevance-based-on-parse-trees/blob/master/src/main/java/opennlp/tools/parse_thicket/kernel_interface/TreeKernelBasedClassifierMultiplePara.java) is the same but the [model files](https://github.com/bgalitsky/relevance-based-on-parse-trees/tree/master/src/test/resources/tree_kernel/TRAINING) vary for the applications below:
+<li>detect security leaks
+<li>detect argumentation
+<li>detect low cohesiveness in text
+<li>detect authors\u2019 doubt and low confidence
+<li>detect fake review
+
+Document classification to six major classes {finance, business, legal, computing, engineering, health} is available via [nearest neighbor model](https://github.com/bgalitsky/relevance-based-on-parse-trees/tree/master/src/main/java/opennlp/tools/doc_classifier/DocClassifier.java). A Lucene training model (1G file) is obtained from Wikipedia corpus. This classifier can be trained for an arbitrary classes once respective Wiki pages are selected and respective [Lucene index is built](https://github.com/bgalitsky/relevance-based-on-parse-trees/tree/master/src/main/java/opennlp/tools/doc_classifier/ClassifierTrainingSetIndexer.java). Once proper training documents are selected from Wikipedia with adequate coverage, the accuracy is usually higher than what can be achieved by word2vec classification models.
+
+### General-purpose [deterministic inductive learner](https://github.com/bgalitsky/relevance-based-on-parse-trees/tree/master/src/main/java/opennlp/tools/jsmlearning) implements JS Mills method of induction and abduction (deduction is also partially implemented).
+
+ Inductive learning implemented as a base for syntactic tree-based learning is similar to the family of approaches such as Explanation-based Learning and Inductive Logic Programming.
+ 
+#### Tree-kernel learning 
+ 
+ is integrated to allow application of SVM learning to sentence-level and paragraph-level linguistic data including discourse. Unlike learning in numerical space, each dimension in tree kernel learning is an occurrence of a particular subtree. Similarity is not a numerical distance but a count of common subtrees. A set of parse trees for individual sentences to represent a paragraph is called
+ [parse thicket](https://en.wikipedia.org/wiki/Parse_Thicket). Its representation as a graph is coded in a tree representation via parenthesis such as [model*.txt] (https://github.com/bgalitsky/relevance-based-on-parse-trees/blob/master/src/test/resources/tree_kernel/model_pos_neg_sentiment.txt).
+ To do model building and predictions, C modules are run in [this directory](https://github.com/bgalitsky/relevance-based-on-parse-trees/tree/master/src/test/resources/tree_kernel), so proper choice need to be made: {svm_classify.linux, svm_classify.max, svm_classify.exe, svm_learn.*}. Also, proper run permissions needs to be set for these files.
+ 
+#### Concept learning 
+ 
+  is a branch of deterministic learning which is applied to attribute-value pairs and possesses useful explainability feature, unlike statistical and deep learning. It is fairly useful for data exploration and visualization since all interesting relations can be visualized. 
+    Concept learning covers inductive and abductive learning and also some cases of deduction. Explore [this package](https://github.com/bgalitsky/relevance-based-on-parse-trees/tree/master/src/main/java/opennlp/tools/fca) for the concept learning-related features.
+
+### Filtering results for Speech Recognition based on semantic meaningfulness
+It takes results from a speech-to-text system and subjects them to [filtering]
+(https://github.com/bgalitsky/relevance-based-on-parse-trees/blob/master/src/main/java/opennlp/tools/similarity/apps/SpeechRecognitionResultsProcessor.java). Those recognized candidate words which do not make sense together are filtered out, based on the frequency of co-occurrences found on the web.
+## Related Research
+Here's the link to the book on [question-answering](https://www.amazon.com/Natural-Language-Question-Answering-system/dp/0868039799/ref=sr_1_10?ie=UTF8&qid=1478871097&sr=8-10&keywords=galitsky)
+
+and [research papers](https://scholar.google.com/citations?hl=ru&user=kR_M3HIAAAAJ).
+
+Also the recent [book related to reasoning and linguistics in humans & machines](https://www.amazon.com/Computational-Autism-Human-Computer-Interaction-Galitsky/dp/3319399713)
+
+## Configuring OpenNLP.Similarity component
+
+VerbNet model is included by default, so that the hand-coded meanings of the verb are used when simularity between verb phrases are computed.
+
+To include word2vector model, [download it](https://deeplearning4j.org/) and make sure the following path is valid:
+`resourceDir + "/w2v/GoogleNews-vectors-negative300.bin.gz"`
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/README.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/README.txt b/opennlp-similarity/README.txt
new file mode 100644
index 0000000..b535487
--- /dev/null
+++ b/opennlp-similarity/README.txt
@@ -0,0 +1,138 @@
+Apache OpenNLP ${pom.version}
+===============================
+
+
+Building from the Source Distribution
+-------------------------------------
+
+At least Maven 3.0.0 is required for building.
+
+To build everything go into the opennlp directory and run the following command:
+    mvn clean install
+   
+The results of the build will be placed  in:
+    opennlp-distr/target/apache-opennlp-[version]-bin.tar-gz (or .zip)
+
+What is in Similarity component in Apache OpenNLP ${pom.version}
+---------------------------------------
+SIMILARITY COMPONENT of OpenNLP
+
+1. Introduction
+This component does text relevance assessment. It takes two portions of texts (phrases, sentences, paragraphs) and returns a similarity score.
+Similarity component can be used on top of search to improve relevance, computing similarity score between a question and all search results (snippets). 
+Also, this component is useful for web mining of images, videos, forums, blogs, and other media with textual descriptions. Such applications as content generation 
+and filtering meaningless speech recognition results are included in the sample applications of this component.
+   Relevance assessment is based on machine learning of syntactic parse trees (constituency trees, http://en.wikipedia.org/wiki/Parse_tree). 
+The similarity score is calculated as the size of all maximal common sub-trees for sentences from a pair of texts (
+www.aaai.org/ocs/index.php/WS/AAAIW11/paper/download/3971/4187, www.aaai.org/ocs/index.php/FLAIRS/FLAIRS11/paper/download/2573/3018,
+www.aaai.org/ocs/index.php/SSS/SSS10/paper/download/1146/1448).
+   The objective of Similarity component is to give an application engineer as tool for text relevance which can be used as a black box, no need to understand 
+ computational linguistics or machine learning. 
+ 
+ 2. Installation
+ Please refer to OpenNLP installation instructions
+ 
+ 3. First use case of Similarity component: search
+ 
+ To start with this component, please refer to SearchResultsProcessorTest.java in package opennlp.tools.similarity.apps
+   public void testSearchOrder() runs web search using Bing API and improves search relevance.
+   Look at the code of 
+      public List<HitBase> runSearch(String query) 
+   and then at 
+      private	BingResponse calculateMatchScoreResortHits(BingResponse resp, String searchQuery)
+   which gets search results from Bing and re-ranks them based on computed similarity score.
+ 
+   The main entry to Similarity component is 
+    SentencePairMatchResult matchRes = sm.assessRelevance(snapshot, searchQuery);
+    where we pass the search query and the snapshot and obtain the similarity assessment structure which includes the similarity score.
+   
+   To run this test you need to obtain search API key from Bing at www.bing.com/developers/s/APIBasics.html and specify it in public class BingQueryRunner in
+  protected static final String APP_ID. 
+  
+  4. Solving a unique problem: content generation
+  To demonstrate the usability of Similarity component to tackle a problem which is hard to solve without a linguistic-based technology, 
+  we introduce a content generation component:
+   RelatedSentenceFinder.java
+   
+   The entry point here is the function call
+   hits = f.generateContentAbout("Albert Einstein");
+   which writes a biography of Albert Einstein by finding sentences on the web about various kinds of his activities (such as 'born', 'graduate', 'invented' etc.).
+   The key here is to compute similarity between the seed expression like "Albert Einstein invented relativity theory" and search result like 
+   "Albert Einstein College of Medicine | Medical Education | Biomedical ...
+    www.einstein.yu.edu/Albert Einstein College of Medicine is one of the nation's premier institutions for medical education, ..."
+    and filter out irrelevant search results.
+   
+   This is done in function 
+   public HitBase augmentWithMinedSentencesAndVerifyRelevance(HitBase item, String originalSentence,
+			List<String> sentsAll)
+			
+   	  SentencePairMatchResult matchRes = sm.assessRelevance(pageSentence + " " + title, originalSentence);
+   You can consult the results in gen.txt, where an essay on Einstein bio is written.
+   
+   These are examples of generated articles, given the article title
+     http://www.allvoices.com/contributed-news/9423860/content/81937916-ichie-sings-jazz-blues-contemporary-tunes
+     http://www.allvoices.com/contributed-news/9415063-britney-spears-femme-fatale-in-north-sf-bay-area
+     
+  5. Solving a high-importance problem: filtering out meaningless speech recognition results.
+  Speech recognitions SDKs usually produce a number of phrases as results, such as 
+  			 "remember to buy milk tomorrow from trader joes",
+			 "remember to buy milk tomorrow from 3 to jones"
+  One can see that the former is meaningful, and the latter is meaningless (although similar in terms of how it is pronounced).
+  We use web mining and Similarity component to detect a meaningful option (a mistake caused by trying to interpret meaningless 
+  request by a query understanding system such as Siri for iPhone can be costly).
+ 
+  SpeechRecognitionResultsProcessor.java does the job:
+  public List<SentenceMeaningfullnessScore> runSearchAndScoreMeaningfulness(List<String> sents)
+  re-ranks the phrases in the order of decrease of meaningfulness.
+  
+  6. Similarity component internals
+  in the package   opennlp.tools.textsimilarity.chunker2matcher
+  ParserChunker2MatcherProcessor.java does parsing of two portions of text and matching the resultant parse trees to assess similarity between 
+  these portions of text.
+  To run ParserChunker2MatcherProcessor
+     private static String MODEL_DIR = "resources/models";
+  needs to be specified
+  
+  The key function
+  public SentencePairMatchResult assessRelevance(String para1, String para2)
+  takes two portions of text and does similarity assessment by finding the set of all maximum common subtrees 
+  of the set of parse trees for each portion of text
+  
+  It splits paragraphs into sentences, parses them, obtained chunking information and produces grouped phrases (noun, evrn, prepositional etc.):
+  public synchronized List<List<ParseTreeChunk>> formGroupedPhrasesFromChunksForPara(String para)
+  
+  and then attempts to find common subtrees:
+  in ParseTreeMatcherDeterministic.java
+		List<List<ParseTreeChunk>> res = md.matchTwoSentencesGroupedChunksDeterministic(sent1GrpLst, sent2GrpLst)
+  
+  Phrase matching functionality is in package opennlp.tools.textsimilarity;
+  ParseTreeMatcherDeterministic.java:
+  Here's the key matching function which takes two phrases, aligns them and finds a set of maximum common sub-phrase
+  public List<ParseTreeChunk> generalizeTwoGroupedPhrasesDeterministic
+  
+  7. Package structure
+  	opennlp.tools.similarity.apps : 3 main applications
+	opennlp.tools.similarity.apps.utils: utilities for above applications
+	
+	opennlp.tools.textsimilarity.chunker2matcher: parser which converts text into a form for matching parse trees
+	opennlp.tools.textsimilarity: parse tree matching functionality
+	
+
+
+
+Requirements
+------------
+Java 1.5 is required to run OpenNLP
+Maven 3.0.0 is required for building it
+
+Known OSGi Issues
+------------
+In an OSGi environment the following things are not supported:
+- The coreference resolution component
+- The ability to load a user provided feature generator class
+
+Note
+----
+The current API contains still many deprecated methods, these
+will be removed in one of our next releases, please
+migrate to our new API.

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/lib/edu.mit.jverbnet-1.2.0.jar
----------------------------------------------------------------------
diff --git a/opennlp-similarity/lib/edu.mit.jverbnet-1.2.0.jar b/opennlp-similarity/lib/edu.mit.jverbnet-1.2.0.jar
new file mode 100644
index 0000000..eba3a97
Binary files /dev/null and b/opennlp-similarity/lib/edu.mit.jverbnet-1.2.0.jar differ

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/lib/ejml-0.23.jar
----------------------------------------------------------------------
diff --git a/opennlp-similarity/lib/ejml-0.23.jar b/opennlp-similarity/lib/ejml-0.23.jar
new file mode 100644
index 0000000..60a37df
Binary files /dev/null and b/opennlp-similarity/lib/ejml-0.23.jar differ

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/lib/javax.json.jar
----------------------------------------------------------------------
diff --git a/opennlp-similarity/lib/javax.json.jar b/opennlp-similarity/lib/javax.json.jar
new file mode 100644
index 0000000..09967d8
Binary files /dev/null and b/opennlp-similarity/lib/javax.json.jar differ

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/lib/joda-time.jar
----------------------------------------------------------------------
diff --git a/opennlp-similarity/lib/joda-time.jar b/opennlp-similarity/lib/joda-time.jar
new file mode 100644
index 0000000..b2aca95
Binary files /dev/null and b/opennlp-similarity/lib/joda-time.jar differ

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/lib/jollyday.jar
----------------------------------------------------------------------
diff --git a/opennlp-similarity/lib/jollyday.jar b/opennlp-similarity/lib/jollyday.jar
new file mode 100644
index 0000000..a6bf8b3
Binary files /dev/null and b/opennlp-similarity/lib/jollyday.jar differ

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/lib/xom.jar
----------------------------------------------------------------------
diff --git a/opennlp-similarity/lib/xom.jar b/opennlp-similarity/lib/xom.jar
new file mode 100644
index 0000000..4eb88da
Binary files /dev/null and b/opennlp-similarity/lib/xom.jar differ

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/com.zip
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/com.zip b/opennlp-similarity/src/main/java/com.zip
new file mode 100644
index 0000000..6f96f23
Binary files /dev/null and b/opennlp-similarity/src/main/java/com.zip differ

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/apps/object_dedup/DedupResult.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/apps/object_dedup/DedupResult.java b/opennlp-similarity/src/main/java/opennlp/tools/apps/object_dedup/DedupResult.java
new file mode 100644
index 0000000..c2a2c0e
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/apps/object_dedup/DedupResult.java
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.apps.object_dedup;
+
+public class DedupResult {
+
+	public DedupResult(String string, int i, boolean b) {
+		// TODO Auto-generated constructor stub
+	}
+
+	public Boolean isDecision() {
+		// TODO Auto-generated method stub
+		return null;
+	}
+
+}


[47/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/TreeKernelBasedClassifier.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/TreeKernelBasedClassifier.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/TreeKernelBasedClassifier.java
new file mode 100644
index 0000000..d1c80ad
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/TreeKernelBasedClassifier.java
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.parse_thicket.kernel_interface;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.logging.Logger;
+
+import org.apache.commons.io.FileUtils;
+
+
+import org.apache.tika.Tika;
+import org.apache.tika.exception.TikaException;
+
+import opennlp.tools.jsmlearning.ProfileReaderWriter;
+import opennlp.tools.parse_thicket.ParseThicket;
+import opennlp.tools.parse_thicket.VerbNetProcessor;
+import opennlp.tools.parse_thicket.apps.MultiSentenceSearchResultsProcessor;
+import opennlp.tools.parse_thicket.matching.Matcher;
+
+public class TreeKernelBasedClassifier {
+	protected static Logger LOG = Logger
+			.getLogger("opennlp.tools.similarity.apps.TreeKernelBasedClassifier");
+	protected ArrayList<File> queuePos = new ArrayList<File>(), queueNeg = new ArrayList<File>();
+  
+	protected Matcher matcher = new Matcher();
+	protected TreeKernelRunner tkRunner = new TreeKernelRunner();
+	protected TreeExtenderByAnotherLinkedTree treeExtender = new TreeExtenderByAnotherLinkedTree();
+
+
+	protected String path;
+	public void setKernelPath (String path){
+		this.path=path;
+	}
+	protected static final String modelFileName = "model.txt";
+
+	protected static final String trainingFileName = "training.txt";
+
+	protected static final String unknownToBeClassified = "unknown.txt";
+
+	protected static final String classifierOutput = "classifier_output.txt";
+	protected static final Float MIN_SVM_SCORE_TOBE_IN = 0.2f;
+	
+	/* main entry point to SVM TK classifier
+     * gets a file, reads it outside of CI, extracts longer paragraphs and builds parse thickets for them.
+     * Then parse thicket dump is processed by svm_classify
+     */
+	public Boolean classifyText(File f){
+		FileUtils.deleteQuietly(new File(path+unknownToBeClassified)); 
+		if (!(new File(path+modelFileName).exists())){
+			LOG.severe("Model file '" +modelFileName + "'is absent: skip SVM classification");
+			return null;
+		}
+		Map<Integer, Integer> countObject = new HashMap<Integer, Integer>(); 
+		int itemCount=0, objectCount = 0;
+		List<String> treeBankBuffer = new ArrayList<String>();	
+		List<String> texts=DescriptiveParagraphFromDocExtractor.getLongParagraphsFromFile(f);
+		List<String> lines = formTreeKernelStructuresMultiplePara(texts, "0");
+		for(String l: lines){
+			countObject.put(itemCount, objectCount);
+			itemCount++;
+		}
+		objectCount++;
+		treeBankBuffer.addAll(lines);		
+
+		// write the lists of samples to a file
+		try {
+			FileUtils.writeLines(new File(path+unknownToBeClassified), null, treeBankBuffer);
+		} catch (IOException e) {
+			LOG.severe("Problem creating parse thicket files '"+ path+unknownToBeClassified + "' to be classified\n"+ e.getMessage() );
+		}
+
+		tkRunner.runClassifier(path, unknownToBeClassified, modelFileName, classifierOutput);
+		// read classification results
+		List<String[]> classifResults = ProfileReaderWriter.readProfiles(path+classifierOutput, ' ');
+
+
+		itemCount=0; objectCount = 0;
+		int currentItemCount=0;
+		float accum = 0;
+		LOG.info("\nsvm scores per paragraph: " );
+		for(String[] line: classifResults){
+			Float val = Float.parseFloat(line[0]);
+			System.out.print(val+" ");
+			accum+=val;
+			currentItemCount++;
+		}
+
+		float averaged = accum/(float)currentItemCount;
+		LOG.info("\n average = "+averaged);
+		currentItemCount=0;
+		Boolean in = false;
+		if (averaged> MIN_SVM_SCORE_TOBE_IN)
+			return true;
+		else
+			return false;
+	}
+
+	protected void addFilesPos(File file) {
+
+		if (!file.exists()) {
+			System.out.println(file + " does not exist.");
+		}
+		if (file.isDirectory()) {
+			for (File f : file.listFiles()) {
+				//if (!(f.getName().endsWith(".txt") || f.getName().endsWith(".pdf")))
+				//	continue;
+				addFilesPos(f);
+				System.out.println(f.getName());
+			}
+		} else {
+			queuePos.add(file);
+		}
+	}
+	
+	protected void addFilesNeg(File file) {
+
+		if (!file.exists()) {
+			System.out.println(file + " does not exist.");
+		}
+		if (file.isDirectory()) {
+			for (File f : file.listFiles()) {
+				//if (!(f.getName().endsWith(".txt")||f.getName().endsWith(".pdf")))
+				//	continue;
+				addFilesNeg(f);
+				System.out.println(f.getName());
+			}
+		} else {
+			queueNeg.add(file);
+		}
+	}
+
+	protected void trainClassifier(
+			String posDirectory, String negDirectory) {
+		
+		queuePos.clear(); queueNeg.clear();
+		addFilesPos(new File(posDirectory));
+		addFilesNeg(new File(negDirectory));
+		
+		List<File> filesPos = new ArrayList<File>(queuePos), filesNeg = new ArrayList<File>(queueNeg);
+		
+		List<String[]> treeBankBuffer = new ArrayList<String[]>();
+
+		for (File f : filesPos) {
+			// get first paragraph of text
+			String text=DescriptiveParagraphFromDocExtractor.getFirstParagraphFromFile(f);		
+			treeBankBuffer.add(new String[]{formTreeKernelStructure(text, "1")});		
+		}	
+		for (File f : filesNeg) {
+			// get first paragraph of text
+			String text=DescriptiveParagraphFromDocExtractor.getFirstParagraphFromFile(f);
+			treeBankBuffer.add(new String[]{formTreeKernelStructure(text, "-1")});		
+		}	
+		
+		// write the lists of samples to a file
+		ProfileReaderWriter.writeReport(treeBankBuffer, path+trainingFileName, ' ');
+		// build the model
+		tkRunner.runLearner(path, trainingFileName, modelFileName);
+	}
+
+	public List<String[]> classifyFilesInDirectory(String dirFilesToBeClassified){
+		List<String[]> treeBankBuffer = new ArrayList<String[]>();
+		queuePos.clear();
+		addFilesPos(new File( dirFilesToBeClassified));
+		List<File> filesUnkn = new ArrayList<File>(queuePos);
+		for (File f : filesUnkn) {	
+			String text=DescriptiveParagraphFromDocExtractor.getFirstParagraphFromFile(f);
+			String line = formTreeKernelStructure(text, "0");
+			treeBankBuffer.add(new String[]{line});		
+		}	
+	
+		// form a file from the texts to be classified
+		ProfileReaderWriter.writeReport(treeBankBuffer, path+unknownToBeClassified, ' ');
+		
+		tkRunner.runClassifier(path, unknownToBeClassified, modelFileName, classifierOutput);
+		// read classification results
+		List<String[]> classifResults = ProfileReaderWriter.readProfiles(path+classifierOutput, ' ');
+		// iterate through classification results and set them as scores for hits
+		List<String[]>results = new ArrayList<String[]>();
+		int count=0;
+		for(String[] line: classifResults){
+			Float val = Float.parseFloat(line[0]);
+			Boolean in = false;
+			if (val> MIN_SVM_SCORE_TOBE_IN)
+				in = true;
+			
+			String[] rline = new String[]{filesUnkn.get(count).getName(), in.toString(), line[0], filesUnkn.get(count).getAbsolutePath() }; // treeBankBuffer.get(count).toString() };
+			results.add(rline);
+			count++;
+			
+		}
+		return results;
+
+	}
+
+	protected List<String> formTreeKernelStructuresMultiplePara(List<String> texts, String flag) {
+		List<String> extendedTreesDumpTotal = new ArrayList<String>();
+		try {
+
+			for(String text: texts){
+				// get the parses from original documents, and form the training dataset
+				LOG.info("About to build pt from "+text);
+				ParseThicket pt = matcher.buildParseThicketFromTextWithRST(text);
+				LOG.info("About to build extended forest ");
+				List<String> extendedTreesDump = treeExtender.buildForestForCorefArcs(pt);
+				for(String line: extendedTreesDump)
+					extendedTreesDumpTotal.add(flag + " |BT| "+line + " |ET| ");
+				LOG.info("DONE");
+			}
+
+		} catch (Exception e) {
+			LOG.severe("Problem forming  parse thicket flat file to be classified\n"+ e.getMessage() );
+		}
+		return extendedTreesDumpTotal;
+	}
+	protected String formTreeKernelStructure(String text, String flag) {
+		String treeBankBuffer = "";
+		try {
+			// get the parses from original documents, and form the training dataset
+			LOG.info("About to build pt from "+text);
+			ParseThicket pt = matcher.buildParseThicketFromTextWithRST(text);
+			LOG.info("About to build extended forest ");
+			List<String> extendedTreesDump = treeExtender.buildForestForCorefArcs(pt);
+			LOG.info("DONE");
+
+			treeBankBuffer+=flag;
+			// form the list of training samples
+			for(String t: extendedTreesDump ){
+				if (BracesProcessor.isBalanced(t))
+					treeBankBuffer+=" |BT| "+t;
+				else
+					System.err.println("Wrong tree: " + t);
+			}
+			if (extendedTreesDump.size()<1)
+				treeBankBuffer+=" |BT| ";
+		} catch (Exception e) {
+			e.printStackTrace();
+		}
+		return treeBankBuffer+ " |ET|";
+	}
+
+	public static void main(String[] args){
+		VerbNetProcessor p = VerbNetProcessor.
+				getInstance("/Users/borisgalitsky/Documents/workspace/deepContentInspection/src/test/resources"); 
+				
+		TreeKernelBasedClassifier proc = new TreeKernelBasedClassifier();
+		proc.setKernelPath("/Users/borisgalitsky/Documents/tree_kernel/");
+		proc.trainClassifier(args[0], args[1]);
+		List<String[]>res = proc.classifyFilesInDirectory(args[2]);
+		ProfileReaderWriter.writeReport(res, "svmDesignDocReport03minus.csv");
+	}
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/TreeKernelBasedClassifierMultiplePara.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/TreeKernelBasedClassifierMultiplePara.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/TreeKernelBasedClassifierMultiplePara.java
new file mode 100644
index 0000000..45fb98c
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/TreeKernelBasedClassifierMultiplePara.java
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.parse_thicket.kernel_interface;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.logging.Logger;
+
+import org.apache.commons.io.FileUtils;
+
+
+import org.apache.tika.Tika;
+import org.apache.tika.exception.TikaException;
+
+import opennlp.tools.jsmlearning.ProfileReaderWriter;
+import opennlp.tools.parse_thicket.ParseThicket;
+import opennlp.tools.parse_thicket.VerbNetProcessor;
+import opennlp.tools.parse_thicket.apps.MultiSentenceSearchResultsProcessor;
+import opennlp.tools.parse_thicket.matching.Matcher;
+
+public class TreeKernelBasedClassifierMultiplePara extends TreeKernelBasedClassifier{
+	boolean bShortRun = false;
+	public void setShortRun(){
+		bShortRun = true;
+	}
+
+
+	public void trainClassifier(
+			String posDirectory, String negDirectory) {
+
+		queuePos.clear(); queueNeg.clear();
+		addFilesPos(new File(posDirectory));
+		addFilesNeg(new File(negDirectory));
+
+		List<File> filesPos = new ArrayList<File>(queuePos), filesNeg = new ArrayList<File>(queueNeg);
+
+		Collection<String> treeBankBuffer = new ArrayList<String>();
+		int countPos=0, countNeg=0;
+
+		for (File f : filesPos) {
+			// get first paragraph of text
+			List<String> texts=DescriptiveParagraphFromDocExtractor.getLongParagraphsFromFile(f);		
+			List<String> lines = formTreeKernelStructuresMultiplePara(texts, "1");
+			treeBankBuffer.addAll(lines);		
+			if (bShortRun && countPos>3000)
+				break;
+
+			countPos++;
+		}	
+		for (File f : filesNeg) {
+			// get first paragraph of text 
+			List<String> texts=DescriptiveParagraphFromDocExtractor.getLongParagraphsFromFile(f);	
+			List<String> lines = formTreeKernelStructuresMultiplePara(texts, "-1");
+			treeBankBuffer.addAll(lines);	
+			if (bShortRun && countNeg>3000)
+				break;
+
+			countNeg++;
+		}	
+
+		// write the lists of samples to a file
+		try {
+			FileUtils.writeLines(new File(path+trainingFileName), null, treeBankBuffer);
+		} catch (IOException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		}
+		//	ProfileReaderWriter.writeReport(treeBankBuffer, path+trainingFileName, ' ');
+		// build the model
+		tkRunner.runLearner(path, trainingFileName, modelFileName);
+	}
+
+	public List<String[]> classifyFilesInDirectory(String dirFilesToBeClassified){
+		Map<Integer, Integer> countObject = new HashMap<Integer, Integer>(); 
+		int itemCount=0, objectCount = 0;
+		List<String> treeBankBuffer = new ArrayList<String>();
+		queuePos.clear();
+		addFilesPos(new File( dirFilesToBeClassified));
+		List<File> filesUnkn = new ArrayList<File>(queuePos);
+		for (File f : filesUnkn) {	
+			List<String> texts=DescriptiveParagraphFromDocExtractor.getLongParagraphsFromFile(f);
+			List<String> lines = formTreeKernelStructuresMultiplePara(texts, "0");
+			for(String l: lines){
+				countObject.put(itemCount, objectCount);
+				itemCount++;
+			}
+			objectCount++;
+			treeBankBuffer.addAll(lines);		
+		}	
+		// write the lists of samples to a file
+		try {
+			FileUtils.writeLines(new File(path+unknownToBeClassified), null, treeBankBuffer);
+		} catch (IOException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		}
+
+		tkRunner.runClassifier(path, unknownToBeClassified, modelFileName, classifierOutput);
+		// read classification results
+		List<String[]> classifResults = ProfileReaderWriter.readProfiles(path+classifierOutput, ' ');
+		// iterate through classification results and set them as scores for hits
+		List<String[]>results = new ArrayList<String[]>();
+
+		itemCount=0; objectCount = 0;
+		int currentItemCount=0;
+		float accum = 0;
+		for(String[] line: classifResults){
+			Float val = Float.parseFloat(line[0]);
+			accum+=val;
+			// last line
+			Boolean bLastLine = false;
+			if (itemCount==classifResults.size()-1)
+				bLastLine = true;
+
+			if (objectCount== countObject .get(itemCount) /*&& !bLastLine*/){
+				itemCount++; 
+				currentItemCount++;
+				continue;
+			}
+			else while(objectCount!= countObject .get(itemCount)-1){
+				objectCount++;
+				String[] rline = new String[]{filesUnkn.get(objectCount).getName(), "unknown", "0",
+						filesUnkn.get(objectCount).getAbsolutePath() , new Integer(itemCount).toString(), new Integer(objectCount).toString()}; 
+				results.add(rline);
+			}
+			objectCount = countObject.get(itemCount);
+			itemCount++; 
+
+			float averaged = accum/(float)currentItemCount;
+			currentItemCount=0;
+			Boolean in = false;
+			if (averaged> MIN_SVM_SCORE_TOBE_IN)
+				in = true;
+
+			String[] rline = new String[]{filesUnkn.get(objectCount).getName(), in.toString(), new Float(averaged).toString(),
+					filesUnkn.get(objectCount).getAbsolutePath() , new Integer(itemCount).toString(), new Integer(objectCount).toString()}; 
+			results.add(rline);
+			accum=0;
+		}
+		return results;
+
+	}
+
+
+	protected List<String> formTreeKernelStructuresMultiplePara(List<String> texts, String flag) {
+		List<String> extendedTreesDumpTotal = new ArrayList<String>();
+		try {
+			for(String text: texts){
+				// get the parses from original documents, and form the training dataset
+				System.out.println("About to build pt from "+text);
+				ParseThicket pt = matcher.buildParseThicketFromTextWithRST(text);
+				System.out.print("About to build extended forest ");
+				List<String> extendedTreesDump = treeExtender.buildForestForCorefArcs(pt);
+				for(String line: extendedTreesDump)
+					extendedTreesDumpTotal.add(flag + " |BT| "+line + " |ET| ");
+				System.out.println("DONE");
+			}
+
+		} catch (Exception e) {
+			e.printStackTrace();
+		}
+		return extendedTreesDumpTotal;
+	}
+
+	public static void main(String[] args){
+		VerbNetProcessor p = VerbNetProcessor.
+				getInstance("/Users/bgalitsky/Documents/relevance-based-on-parse-trees/src/test/resources"); 
+
+		TreeKernelBasedClassifierMultiplePara proc = new TreeKernelBasedClassifierMultiplePara();
+		proc.setKernelPath("/Users/bgalitsky/Documents/relevance-based-on-parse-trees/src/test/resources/tree_kernel/");
+		proc.trainClassifier(
+
+				"/Users/bgalitsky/Documents/relevance-based-on-parse-trees/src/test/resources/style_recognizer/txt/ted",
+				"/Users/bgalitsky/Documents/relevance-based-on-parse-trees/src/test/resources/style_recognizer/txt/Tedi");
+
+		//		List<String[]>res = proc.classifyFilesInDirectory(args[2]);
+		//		ProfileReaderWriter.writeReport(res, "svmDesignDocReport05plus.csv");
+	}
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/TreeKernelBasedClassifierOfDiscourseTree.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/TreeKernelBasedClassifierOfDiscourseTree.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/TreeKernelBasedClassifierOfDiscourseTree.java
new file mode 100644
index 0000000..71e8245
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/TreeKernelBasedClassifierOfDiscourseTree.java
@@ -0,0 +1,326 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.parse_thicket.kernel_interface;
+
+
+import java.util.ArrayList;
+import java.util.List;
+
+import opennlp.tools.parse_thicket.ParseThicket;
+import opennlp.tools.parse_thicket.VerbNetProcessor;
+import opennlp.tools.parse_thicket.external_rst.MatcherExternalRST;
+import opennlp.tools.parse_thicket.external_rst.ParseThicketWithDiscourseTree;
+
+/*
+ * This class performs TK learning based on parse thicket which includes RST relations only 
+ * based on Surdeanu at al RST parser. It does sentence parsing and NLP pipeline of 
+ * Surdeanu's wrapper of Stanford NLP
+ */
+public class TreeKernelBasedClassifierOfDiscourseTree extends TreeKernelBasedClassifierMultiplePara{
+
+	private MatcherExternalRST matcherRST = new MatcherExternalRST();
+
+	protected List<String> formTreeKernelStructuresMultiplePara(List<String> texts, String flag) {
+		//TODO
+		//this.setShortRun();	
+		List<String> extendedTreesDumpTotal = new ArrayList<String>();
+		try {
+
+			for(String text: texts){
+				// get the parses from original documents, and form the training dataset
+				try {
+					System.out.print("About to build pt with external rst from "+text + "\n...");
+					ParseThicket pt = matcherRST.buildParseThicketFromTextWithRST(text);
+					if (pt == null)
+						continue;
+					System.out.print("About to build extended forest with external rst...");
+					List<String> extendedTreesDump =  // use direct option (true
+							buildReptresentationForDiscourseTreeAndExtensions((ParseThicketWithDiscourseTree)pt, false);
+									//true);
+					for(String line: extendedTreesDump)
+						extendedTreesDumpTotal.add(flag + " |BT| "+line + " |ET| ");
+					System.out.println("DONE");
+				} catch (Exception e) {
+					e.printStackTrace();
+				}
+			}
+		} catch (Exception e) {
+			e.printStackTrace();
+		}
+		return extendedTreesDumpTotal;
+	}
+
+	private List<String> buildReptresentationForDiscourseTreeAndExtensions(ParseThicketWithDiscourseTree pt, boolean bDirectDT){
+		List<String> extendedTreesDump = new ArrayList<String>();
+		if (!bDirectDT)
+			// option 1: use RST relation for extended trees 
+			extendedTreesDump = treeExtender.buildForestForRSTArcs(pt);
+		else {
+			// option 2: use DT directly
+			extendedTreesDump.add(pt.getDtDump());
+		    extendedTreesDump.add(pt.getDtDumpWithPOS());
+		    extendedTreesDump.add(pt.getDtDumpWithEmbeddedTrees());
+		    extendedTreesDump.add(pt.getDtDumpWithVerbNet());
+		}		
+		return extendedTreesDump;
+	}
+	
+	/*
+	 * dtDump
+	 * 1 |BT| (elaboration (joint (attribution (I though) (I d tell you a little about what I like to write )) (joint (And I like to immerse myself in my topics ) (joint (I just like to dive right i) (and become sort of a human guinea pig )))) (elaboration (joint (And I see my life as a series of experiments ) (joint (So , I work for Esquire magazine ) (elaboration (elaboration (and a couple of years ago I wrote an articl) (called My Outsourced Life )) (enablement (where I hired a team of people in Bangalore , India ) (to live my life for me ))))) (elaboration (So they answered my emails ) (They answered my phone )))) |ET|
+	 * 
+	 * getDtDumpWithPOS
+	 * 
+	 *  1 |BT| (elaboration (joint (attribution (I PRP)(thought VBD) (I PRP)(d NN)(tell VBP)(you PRP)(a DT)(little JJ)(about IN)(what WP)(I PRP)(like VBP)(to TO)(write VB)) (joint (And CC)(I PRP)(like VBP)(to TO)(immerse VB)(myself PRP)(in IN)(my PRP$)(topics NNS) (joint (I PRP)(just RB)(like VBP)(to TO)(dive NN)(right NN)(in IN) (and CC)(become VB)(sort NN)(of IN)(a DT)(human JJ)(guinea NN)(pig NN)))) (elaboration (joint (And CC)(I PRP)(see VBP)(my PRP$)(life NN)(as IN)(a DT)(series NN)(of IN)(experiments NNS) (joint (So RB)(I PRP)(work VBP)(for IN)(Esquire NNP)(magazine NN) (elaboration (elaboration (and CC)(a DT)(couple NN)(of IN)(years NNS)(ago IN)(I PRP)(wrote VBD)(an DT)(article NN) (called VBN)(My PRP$)(Outsourced JJ)(Life NNP)) (enablement (where WRB)(I PRP)(hired VBD)(a DT)(team NN)(of IN)(people NNS)(in IN)(Bangalore NNP)(India NNP) (to TO)(live VB)(my PRP$)(life NN)(for IN)(me PRP))))) (elaboration (So IN)(they PRP)(answered VBD)(my PRP$)(emails NNS) (They PRP)(answered VBD)
 (my PRP$)(phone NN)))) |ET| 
+	 * 
+	 * getDtDumpWithEmbeddedTrees()
+	 * 1 |BT| (elaboration (joint (attribution (SBAR (S (NP (PRP I)) (VP (ADVP (NN d)) (VBP tell) (NP (PRP you)) (PP (NP (DT a) (JJ little)) (IN about) (SBAR (WHNP (WP what)) (S (NP (PRP I)) (VP (VBP like) (S (VP (TO to) (VP (VB write))))))))))) (VBP tell)) (joint (VP (VBP like) (S (VP (TO to) (VP (VB immerse) (NP (PRP myself)) (PP (IN in) (NP (PRP$ my) (NNS topics))))))) (joint (VP (VP (VBP like) (PP (TO to) (NP (NN dive) (NN right))) (PP (IN in))) (CC and) (VP (VB become) (NP (NP (NN sort)) (PP (IN of) (NP (DT a) (JJ human) (NN guinea) (NN pig)))))) (NP (NP (NN sort)) (PP (IN of) (NP (DT a) (JJ human) (NN guinea) (NN pig))))))) (elaboration (joint (VP (VBP see) (NP (PRP$ my) (NN life)) (PP (IN as) (NP (NP (DT a) (NN series)) (PP (IN of) (NP (NNS experiments)))))) (joint (S (NP (PRP I)) (VP (VBP work) (PP (IN for) (NP (NNP Esquire) (NN magazine))))) (elaboration (elaboration (NN couple) (JJ Outsourced)) (enablement (VP (VBP work) (PP (IN for) (NP (NNP Esquire) (NN magazine)))) (NP (PR
 P$ my) (NN life)))))) (elaboration (VP (VBD answered) (NP (PRP$ my) (NNS emails))) (NP (PRP$ my) (NN phone))))) |ET|
+	 
+	 pt.getDtDumpWithVerbNet()
+	 1 |BT| (elaboration (joint (attribution (I PRP)(thought VBD) (I PRP)(d NN) (tell  (tell-372 tell-372 tell-372 ) (NP V NP NP V NP PP-topic NP V NP S ) (NP NP-PPof-PP NP-S ) ) (you PRP)(a DT)(little JJ)(about IN)(what WP)(I PRP)(like VBP)(to TO)(write VB)) (joint (And CC)(I PRP)(like VBP)(to TO)(immerse VB)(myself PRP)(in IN)(my PRP$)(topics NNS) (joint (I PRP)(just RB)(like VBP)(to TO)(dive NN)(right NN)(in IN) (and CC)(become VB)(sort NN)(of IN)(a DT)(human JJ)(guinea NN)(pig NN)))) (elaboration (joint (And CC)(I PRP) (see  (see-301 see-301 see-301 ) (NP V NP NP V that S NP V NP-ATTR-POS PP-oblique NP V how S NP V what S ) (Basic Transitive S Attribute Object Possessor-Attribute Factoring Alternation HOW-S WHAT-S ) ) (my PRP$)(life NN)(as IN)(a DT)(series NN)(of IN)(experiments NNS) (joint (So RB)(I PRP)(work VBP)(for IN)(Esquire NNP)(magazine NN) (elaboration (elaboration (and CC)(a DT)(couple NN)(of IN)(years NNS)(ago IN)(I PRP)(wrote VBD)(an DT)(article NN) (call  (dub-293 dub-
 293 dub-293 ) (NP V NP NP NP V NP ) (NP-NP Basic Transitive ) ) (My PRP$)(Outsourced JJ)(Life NNP)) (enablement (where WRB)(I PRP) (hire  (hire-1353 hire-1353 hire-1353 ) (NP V NP NP V NP PP-predicate ) (NP NP-PPas-PP ) ) (a DT)(team NN)(of IN)(people NNS)(in IN)(Bangalore NNP)(India NNP) (to TO)(live VB)(my PRP$)(life NN)(for IN)(me PRP))))) (elaboration (So IN)(they PRP)(answered VBD)(my PRP$)(emails NNS) (They PRP)(answered VBD)(my PRP$)(phone NN)))) |ET|
+	 *
+	 */
+	
+	
+	
+	public static void main(String[] args){
+		VerbNetProcessor p = VerbNetProcessor.
+				getInstance("/Users/borisgalitsky/Documents/workspace/relevance-based-on-parse-trees/src/test/resources"); 
+
+		TreeKernelBasedClassifierOfDiscourseTree proc = new TreeKernelBasedClassifierOfDiscourseTree();
+		proc.setKernelPath("/Users/borisgalitsky/Documents/workspace/relevance-based-on-parse-trees/src/test/resources/tree_kernel/");
+		proc.trainClassifier(
+//				"/Users/bgalitsky/Documents/relevance-based-on-parse-trees/src/test/resources/style_recognizer/txt/ted",
+//				"/Users/bgalitsky/Documents/relevance-based-on-parse-trees/src/test/resources/style_recognizer/txt/Tedi");
+				//"/Users/bgalitsky/Documents/ENRON/enron_random",
+				
+	//			"/Users/bgalitsky/Documents/ENRON/data11_17",
+	//	"/Users/bgalitsky/Documents/ENRON/enron_secrecy"
+	//			"/Users/bgalitsky/Downloads/op_spam_v1.4/positive_polarity/truthful_from_TripAdvisor",
+				"/Users/bgalitsky/Downloads/op_spam_v1.4/negative_polarity/deceptive_from_MTurk",
+				"/Users/bgalitsky/Downloads/op_spam_v1.4/negative_polarity/truthful_from_Web" 
+				);
+				
+//				"/Users/borisgalitsky/Documents/workspace/relevance-based-on-parse-trees/src/test/resources/style_recognizer/txt/ted",
+//				"/Users/borisgalitsky/Documents/workspace/relevance-based-on-parse-trees/src/test/resources/style_recognizer/txt/Tedi");
+
+	}
+
+}
+/*
+ * 
+RST - based run
+Number of examples: 6980, linear space size: 10
+ted vs Tedi
+
+estimating ...
+Setting default regularization parameter C=1.0000
+Optimizing...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
 .....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
 .....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
 .......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
+ Checking optimality of inactive variables...done.
+ Number of inactive variables = 1931
+done. (3597 iterations)
+Optimization finished (78 misclassified, maxdiff=0.00100).
+Runtime in cpu-seconds: 198.37
+Number of SV: 3830 (including 652 at upper bound)
+L1 loss: loss=261.78883
+Norm of weight vector: |w|=41.37067
+Norm of longest example vector: |x|=1.00000
+Estimated VCdim of classifier: VCdim<=1712.53247
+Computing XiAlpha-estimates...done
+Runtime for XiAlpha-estimates in cpu-seconds: 0.05
+XiAlpha-estimate of the error: error<=11.53% (rho=1.00,depth=0)
+XiAlpha-estimate of the recall: recall=>97.01% (rho=1.00,depth=0)
+XiAlpha-estimate of the precision: precision=>89.47% (rho=1.00,depth=0)
+Number of kernel evaluations: 73092240
+
+GENERAL RUN (the same set of texts)
+Number of examples: 21146, linear space size: 10
+
+estimating ...
+Setting default regularization parameter C=1.0000
+Optimizing...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
 .....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
 .....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
 .....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
 .....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
 ..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
+ Checking optimality of inactive variables...done.
+ Number of inactive variables = 8849
+done. (5770 iterations)
+Optimization finished (231 misclassified, maxdiff=0.00098).
+Runtime in cpu-seconds: 1486.33
+Number of SV: 5368 (including 940 at upper bound)
+L1 loss: loss=582.99311
+Norm of weight vector: |w|=46.91885
+Norm of longest example vector: |x|=1.00000
+Estimated VCdim of classifier: VCdim<=2202.37876
+Computing XiAlpha-estimates...done
+Runtime for XiAlpha-estimates in cpu-seconds: 0.13
+XiAlpha-estimate of the error: error<=5.57% (rho=1.00,depth=0)
+XiAlpha-estimate of the recall: recall=>98.42% (rho=1.00,depth=0)
+XiAlpha-estimate of the precision: precision=>95.18% (rho=1.00,depth=0)
+Number of kernel evaluations: 550748695
+Writing model file...done
+
+
+Number of examples: 7461, linear space size: 10
+
+estimating ...
+Setting default regularization parameter C=1.0000
+Optimizing...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
 .....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
 .....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
 .......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
+ Checking optimality of inactive variables...done.
+ Number of inactive variables = 2091
+done. (3773 iterations)
+Optimization finished (87 misclassified, maxdiff=0.00096).
+Runtime in cpu-seconds: 231.42
+Number of SV: 4092 (including 680 at upper bound)
+L1 loss: loss=280.03696
+Norm of weight vector: |w|=42.82963
+Norm of longest example vector: |x|=1.00000
+Estimated VCdim of classifier: VCdim<=1835.37688
+Computing XiAlpha-estimates...done
+Runtime for XiAlpha-estimates in cpu-seconds: 0.05
+XiAlpha-estimate of the error: error<=11.54% (rho=1.00,depth=0)
+XiAlpha-estimate of the recall: recall=>96.75% (rho=1.00,depth=0)
+XiAlpha-estimate of the precision: precision=>89.59% (rho=1.00,depth=0)
+Number of kernel evaluations: 94432306
+Writing model file...done
+
+
+
+SMALL SET
+
+Number of examples: 172, linear space size: 10
+
+estimating ...
+Setting default regularization parameter C=1.0000
+Optimizing.......................................................done. (56 iterations)
+Optimization finished (0 misclassified, maxdiff=0.00076).
+Runtime in cpu-seconds: 0.01
+Number of SV: 172 (including 59 at upper bound)
+L1 loss: loss=7.38525
+Norm of weight vector: |w|=12.46777
+Norm of longest example vector: |x|=1.00000
+Estimated VCdim of classifier: VCdim<=156.44537
+Computing XiAlpha-estimates...done
+Runtime for XiAlpha-estimates in cpu-seconds: 0.00
+XiAlpha-estimate of the error: error<=44.77% (rho=1.00,depth=0)
+XiAlpha-estimate of the recall: recall=>79.55% (rho=1.00,depth=0)
+XiAlpha-estimate of the precision: precision=>54.26% (rho=1.00,depth=0)
+Number of kernel evaluations: 20139
+Writing model file...done
+
+
+LONGER RUN, DTs only
+Number of examples: 720, linear space size: 10
+
+estimating ...
+Setting default regularization parameter C=1.0000
+Optimizing............................................................................................................................................................................................................................................................................
+ Checking optimality of inactive variables...done.
+ Number of inactive variables = 114
+done. (269 iterations)
+Optimization finished (11 misclassified, maxdiff=0.00096).
+Runtime in cpu-seconds: 0.17
+Number of SV: 712 (including 140 at upper bound)
+L1 loss: loss=117.83422
+Norm of weight vector: |w|=12.73402
+Norm of longest example vector: |x|=1.00000
+Estimated VCdim of classifier: VCdim<=163.15526
+Computing XiAlpha-estimates...done
+Runtime for XiAlpha-estimates in cpu-seconds: 0.00
+XiAlpha-estimate of the error: error<=20.14% (rho=1.00,depth=0)
+XiAlpha-estimate of the recall: recall=>99.14% (rho=1.00,depth=0)
+XiAlpha-estimate of the precision: precision=>80.42% (rho=1.00,depth=0)
+Number of kernel evaluations: 283615
+Writing model file...done
+
+HYBRID RUN
+Number of examples: 8301, linear space size: 10
+
+estimating ...
+Setting default regularization parameter C=1.0000
+Optimizing................................
+ Checking optimality of inactive variables...done.
+ Number of inactive variables = 2323
+done. (4206 iterations)
+Optimization finished (98 misclassified, maxdiff=0.00099).
+Runtime in cpu-seconds: 299.94
+Number of SV: 4870 (including 846 at upper bound)
+L1 loss: loss=398.61389
+Norm of weight vector: |w|=44.95124
+Norm of longest example vector: |x|=1.00000
+Estimated VCdim of classifier: VCdim<=2021.61414
+Computing XiAlpha-estimates...done
+Runtime for XiAlpha-estimates in cpu-seconds: 0.05
+XiAlpha-estimate of the error: error<=12.32% (rho=1.00,depth=0)
+XiAlpha-estimate of the recall: recall=>97.15% (rho=1.00,depth=0)
+XiAlpha-estimate of the precision: precision=>88.53% (rho=1.00,depth=0)
+Number of kernel evaluations: 138447398
+Writing model file...done
+
+HYBRID FULL RUN
+
+Number of examples: 2880, linear space size: 10
+
+estimating ...
+Setting default regularization parameter C=1.0021
+Optimizing...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
 ................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
+ Checking optimality of inactive variables...done.
+ Number of inactive variables = 1035
+done. (1820 iterations)
+Optimization finished (162 misclassified, maxdiff=0.00099).
+Runtime in cpu-seconds: 1.35
+Number of SV: 1552 (including 556 at upper bound)
+L1 loss: loss=426.90789
+Norm of weight vector: |w|=25.52139
+Norm of longest example vector: |x|=1.00000
+Estimated VCdim of classifier: VCdim<=652.34149
+Computing XiAlpha-estimates...done
+Runtime for XiAlpha-estimates in cpu-seconds: 0.01
+XiAlpha-estimate of the error: error<=23.92% (rho=1.00,depth=0)
+XiAlpha-estimate of the recall: recall=>92.67% (rho=1.00,depth=0)
+XiAlpha-estimate of the precision: precision=>80.55% (rho=1.00,depth=0)
+Number of kernel evaluations: 4075095
+Writing model file...done
+
+
+
+
+
+positive vs negative sentiment
+"/Users/bgalitsky/Downloads/op_spam_v1.4/positive_polarity/truthful_from_TripAdvisor",
+				"/Users/bgalitsky/Downloads/op_spam_v1.4/negative_polarity/truthful_from_Web" 
+
+Number of examples: 15930, linear space size: 10
+
+estimating ...
+Setting default regularization parameter C=1.0000
+Optimizing...........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
 .....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
 .....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
 .....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
 .....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
 .....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
 .....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
 .....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
 .....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
 .....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
 .....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
 ............................................................................................................................................................................
+ Checking optimality of inactive variables...done.
+ Number of inactive variables = 4348
+done. (11130 iterations)
+Optimization finished (14 misclassified, maxdiff=0.00098).
+Runtime in cpu-seconds: 2213.21
+Number of SV: 9219 (including 875 at upper bound)
+L1 loss: loss=126.05211
+Norm of weight vector: |w|=71.25103
+Norm of longest example vector: |x|=1.00000
+Estimated VCdim of classifier: VCdim<=5077.70889
+Computing XiAlpha-estimates...done
+Runtime for XiAlpha-estimates in cpu-seconds: 0.09
+XiAlpha-estimate of the error: error<=10.15% (rho=1.00,depth=0)
+XiAlpha-estimate of the recall: recall=>89.36% (rho=1.00,depth=0)
+XiAlpha-estimate of the precision: precision=>89.85% (rho=1.00,depth=0)
+Number of kernel evaluations: 837061668
+Writing model file...done
+ */
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/style_classif/TSNE_ImporterProcessor.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/style_classif/TSNE_ImporterProcessor.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/style_classif/TSNE_ImporterProcessor.java
new file mode 100644
index 0000000..ef00e94
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/kernel_interface/style_classif/TSNE_ImporterProcessor.java
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.parse_thicket.kernel_interface.style_classif;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.util.HashMap;
+import java.util.Map;
+
+import opennlp.tools.parse_thicket.VerbNetProcessor;
+import opennlp.tools.parse_thicket.kernel_interface.TreeKernelBasedClassifierMultiplePara;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang.StringUtils;
+
+public class TSNE_ImporterProcessor {
+	private static String importFilePath = "all-tsne2.txt";
+	public String resourceWorkDir = new File(".").getAbsolutePath().replace("/.", "") + 
+			"/src/test/resources/style_recognizer/";
+
+	public void importFileCreatClassifDirs() {
+		Map<Integer, String> id_Text = new HashMap<Integer, String>();
+		Map<Integer, String> id_Label = new HashMap<Integer, String>();
+
+		try {
+			FileUtils.cleanDirectory(new File(resourceWorkDir+"/txt"));
+		} catch (IOException e2) {
+			e2.printStackTrace();
+		}
+
+		String text = null;
+		try {
+			text = FileUtils.readFileToString(new File(resourceWorkDir+importFilePath ), Charset.defaultCharset().toString());
+		} catch (IOException e) {
+
+			e.printStackTrace();
+		}
+
+		String[] portions = StringUtils.substringsBetween(text, "<text ", "/text>");
+		for(int i=0; i<portions.length; i++){
+			String label = StringUtils.substringBetween(portions[i], "id=\"", "\">");
+			String po =  StringUtils.substringBetween(portions[i],  "\">", "<");
+			id_Text.put(i, po);
+			id_Label.put(i, label);
+			if (true){
+				String localDirName = label.substring(0, 4);
+				if (!new File(resourceWorkDir+"txt/"+localDirName).exists())
+					try {
+						FileUtils.forceMkdir(new File(resourceWorkDir+"txt/"+localDirName));
+					} catch (IOException e1) {
+						e1.printStackTrace();
+					}
+				try {
+					label = label.replace('/', '_');
+					String fullPath = resourceWorkDir+"txt/"+localDirName+"/"+i+label+".txt";
+					FileUtils.writeStringToFile(new File(fullPath), po);
+				} catch (IOException e) {
+					e.printStackTrace();
+				}
+			}
+		}
+
+	}
+
+	public static void main(String[] args){
+		TSNE_ImporterProcessor thisProc = new TSNE_ImporterProcessor();
+		thisProc.importFileCreatClassifDirs();
+
+		VerbNetProcessor p = VerbNetProcessor.
+				getInstance("/Users/borisgalitsky/Documents/workspace/deepContentInspection/src/test/resources"); 
+
+		TreeKernelBasedClassifierMultiplePara proc = new TreeKernelBasedClassifierMultiplePara();
+		proc.setKernelPath("/Users/borisgalitsky/Documents/tree_kernel/");
+		proc.trainClassifier(thisProc.resourceWorkDir+"/txt/Tele", 
+				thisProc.resourceWorkDir+"/txt/Tels");
+		//www.sciencedirect.com/science/article/pii/S095070511300138X
+	}
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/FrameQueryBasedIExtractor.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/FrameQueryBasedIExtractor.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/FrameQueryBasedIExtractor.java
new file mode 100644
index 0000000..f75c0b1
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/FrameQueryBasedIExtractor.java
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.parse_thicket.matching;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import opennlp.tools.parse_thicket.VerbNetProcessor;
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+import opennlp.tools.textsimilarity.TextProcessor;
+
+public class FrameQueryBasedIExtractor {
+	List<GeneralizationResult> templates = new ArrayList<GeneralizationResult>();
+	Matcher matcher = Matcher.getInstance();
+
+
+
+	private void init() {
+		templates.clear();
+
+	}
+	public void buildPTTemplates(String[] smpls){
+
+		GeneralizationResult templateCurr = matcher.assessRelevanceG(smpls[0], smpls[1]);
+		for(int i=2; i<smpls.length; i++){
+
+			templateCurr = matcher.assessRelevanceG(templateCurr, smpls[i]);
+		}
+
+		templates.add(templateCurr);
+		System.out.println("template = "+ templateCurr);
+
+	}
+
+	public void buildTemplates(String[] samples){
+		for(String setOfSamples : samples){
+			List<String> smpls = TextProcessor.splitToSentences(setOfSamples);
+			if (smpls.size()<2)
+				continue;
+			
+			GeneralizationResult templateCurr = matcher.assessRelevanceG(smpls.get(0), smpls.get(1));
+			for(int i=2; i<smpls.size(); i++){
+
+				templateCurr = matcher.assessRelevanceG(templateCurr, smpls.get(i));
+			}
+
+			templates.add(templateCurr);
+			System.out.println("template = "+ templateCurr+ "\n");
+		}
+	}
+	
+	public void buildTemplatesPairWise(String[] samples){
+		for(String setOfSamples : samples){
+			List<String> smpls = TextProcessor.splitToSentences(setOfSamples);
+
+			GeneralizationResult templateCurr = null;
+			for(int i=0; i<smpls.size(); i++)
+				for(int j=i+1; j< smpls.size(); j++){
+					templateCurr = matcher.assessRelevanceG(smpls.get(i), smpls.get(j));
+					templates.add(templateCurr);
+					System.out.println("template = "+ templateCurr+ "\n");
+			}
+		}
+	}
+
+	List<GeneralizationResult>  doIE(String text){
+		List<GeneralizationResult> fires = new ArrayList<GeneralizationResult>();
+
+		List<String> sentences = TextProcessor.splitToSentences(text);{
+			for(String sent: sentences){
+				for(GeneralizationResult t: templates){
+					GeneralizationResult res = matcher.assessRelevanceG(t.getGen(), sent);
+					boolean fire = matcher.isCoveredByTemplate(t.getGen(), res.getGen());
+					System.out.println(res+ " => "+ fire + "\n");
+					if (fire){
+						res.setIfFire(fire);
+						res.setText(sent);
+						fires.add(res);
+						System.out.println("=====================\n TEMPLATE FIRED: "+ sent + "\n====================\n");
+					}
+				}
+			}
+
+		}
+		return fires;
+	}
+
+	List<GeneralizationResult>  doIEforPT(String text){
+		List<GeneralizationResult> fires = new ArrayList<GeneralizationResult>();
+
+		for(GeneralizationResult t: templates){
+			GeneralizationResult res = matcher.assessRelevanceG(t.getGen(), text);
+			boolean fire = matcher.isCoveredByTemplate(t.getGen(), res.getGen());
+			System.out.println(res+ " =PT=> "+ fire + "\n");
+			res.setIfFire(fire);
+			res.setText(text);
+			if (fire)
+				fires.add(res);
+			
+		}
+		return fires;
+	}
+
+
+	public static void main(String[] args){
+		VerbNetProcessor.getInstance("/Users/borisgalitsky/Documents/workspace/opennlp-similarity/src/test/resources");
+		FrameQueryBasedIExtractor extractor = new FrameQueryBasedIExtractor();
+		
+		String[] texts = new String[]{"An amusement park sells adult tickets for $3 and kids tickets for $2, and got the revenue $500 yesterday.",
+						"A certified trainer conducts training for adult customers for $30 per hour and kid customer for $20 per hour, and got the revenue $1000 today."};		
+		extractor.buildPTTemplates(texts);
+		
+		 texts = new String[]{"Crossing the snow slope was dangerous. They informed in the blog that an ice axe should be used. However, I am reporting that crossing the snow field in the late afternoon I had to use crampons.",
+				"I could not cross the snow creek since it was dangerous. This was because the previous hiker reported that ice axe should be used in late afternoon.  To inform the fellow hikers, I had to use crampons going across the show field in the late afternoon ",
+		};		
+		extractor.buildPTTemplates(texts);
+		List<GeneralizationResult>  res = extractor.doIEforPT( "I had to use crampons to cross snow slopes without an ice axe in late afternoon. However in summer I do not feel it was dangerous crossing the snow.");
+
+		System.exit(0);
+
+		extractor.buildTemplates(new String[] { ""
+				+ "A junior sale engineer expert travels to customers on site. A junior design expert goes to customer companies. "
+				+ "A junior software engineer rushes to customer sites. "   
+		});
+		res = extractor.doIE( "Cisco junior sale representative expert flew to customers data centers. ");
+
+		extractor.init();
+
+		extractor.buildTemplates(new String[] { "John Doe send his California driver license 1234567. "
+				+ "Jill Paparapathi received her Ohio license 4567456"   });
+
+		res = extractor.doIE( "Mary Jones send her Canada prisoner id number 666666666. Mary Stewart hid her Mexico cook id number 666666666 . Robin Hood mentioned his UK fisher id  2345."
+				+ "Yesterday Peter Doe hid his Bolivia set id number 666666666. Robin mentioned her best Peru fisher man id  2345. Spain hid her Catalonian driver id number 666666666. John Poppins hid her  prisoner id  666666666. "
+				+ "Microsoft announced its Canada windows azure release number 666666666. John Poppins hid her Apple id  666666666");
+
+	}
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/GeneralizationResult.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/GeneralizationResult.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/GeneralizationResult.java
new file mode 100644
index 0000000..4c8ae6a
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/GeneralizationResult.java
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.parse_thicket.matching;
+
+import java.util.List;
+
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+
+public class GeneralizationResult {
+	List<List<ParseTreeChunk>> gen;
+	Boolean bFire;
+	String text;
+
+	public List<List<ParseTreeChunk>> getGen() {
+		return gen;
+	}
+
+	public void setGen(List<List<ParseTreeChunk>> gen) {
+		this.gen = gen;
+	}
+
+	public GeneralizationResult(List<List<ParseTreeChunk>> gen) {
+
+		this.gen = gen;
+	}
+	
+	public String toString(){
+		return this.gen.toString();
+	}
+
+	public void setIfFire(boolean fire) {
+		this.bFire = fire;
+		
+	}
+
+	public void setText(String text2) {
+		this.text = text2;
+		
+	}
+
+	public Boolean getbFire() {
+		return bFire;
+	}
+
+
+
+	public String getText() {
+		return text;
+	}
+	
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/LemmaGeneralizer.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/LemmaGeneralizer.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/LemmaGeneralizer.java
new file mode 100644
index 0000000..de966e5
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/LemmaGeneralizer.java
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.parse_thicket.matching;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import opennlp.tools.parse_thicket.IGeneralizer;
+import opennlp.tools.stemmer.PStemmer;
+import opennlp.tools.word2vec.W2VDistanceMeasurer;
+
+public class LemmaGeneralizer implements IGeneralizer<String> {
+	public static final String w2vPrefix = "w2v_";
+	PStemmer ps = new PStemmer();
+	String pos = null;
+	W2VDistanceMeasurer w2v = null; 
+	public LemmaGeneralizer() {
+		w2v = W2VDistanceMeasurer.getInstance();
+    }
+
+	public void setPOS(String posToSet){
+		this.pos = posToSet;
+	}
+
+	@Override
+	public List<String> generalize(Object o1, Object o2) {
+		List<String> results = new ArrayList<String>();
+		boolean bEqual = false;
+
+		String lemma1 = (String)o1, lemma2 = (String)o2;
+	
+			
+			lemma1 = lemma1.toLowerCase();
+			lemma2 = lemma2.toLowerCase();
+
+			if (lemma1.equals(lemma2)) {
+				bEqual = true;
+				results.add(lemma1);
+				return results;
+			}
+
+
+			if ((lemma1.equals(lemma2 + "s") || lemma2.equals(lemma1 + "s"))
+					|| lemma1.endsWith(lemma2) || lemma2.endsWith(lemma1)
+					|| lemma1.startsWith(lemma2) || lemma2.startsWith(lemma1)) {
+				bEqual = true;
+				results.add(lemma1);
+				return results;
+			}
+
+			try {
+				if (ps != null) {
+					if (ps.stem(lemma1).toString()
+							.equalsIgnoreCase(ps.stem(lemma2).toString())) {
+						bEqual = true;
+						results.add(lemma1);
+						return results;
+					}
+				}
+			} catch (Exception e) {
+				System.err.println("Problem processing " + lemma1 + " " + lemma2);
+				return results;
+			}
+			// if different words, then compute word2vec distance and write the value as a string
+			if (w2v.vec!=null){
+				double value = w2v.vec.similarity(lemma1,  lemma2);
+				results.add(w2vPrefix+new Float(value).toString());
+			}
+			return results;
+		}
+
+
+
+		
+
+
+
+
+	}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/MyMatcher.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/MyMatcher.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/MyMatcher.java
new file mode 100644
index 0000000..97ab041
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/MyMatcher.java
@@ -0,0 +1,126 @@
+package opennlp.tools.parse_thicket.matching;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.log4j.Logger;
+import org.deeplearning4j.berkeley.Pair;
+
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+import opennlp.tools.textsimilarity.ParseTreeChunkListScorer;
+import opennlp.tools.textsimilarity.TextProcessor;
+
+import org.apache.log4j.Level;
+import org.apache.log4j.LogManager;
+
+import edu.stanford.nlp.util.StringUtils;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileReader;
+import java.util.*;
+
+/**
+ * Created by sanviswa on 10/29/16.
+ */
+public class MyMatcher {
+
+    Matcher m = new Matcher();
+
+
+    public static void main(String[] args) throws Exception
+    {
+
+        MyMatcher myMatcher = new MyMatcher();
+        myMatcher.runTest(myMatcher.readFile());
+    }
+
+    public List<String> readFile() throws Exception
+    {
+     //   BufferedReader br = new BufferedReader(new FileReader(this.getClass().getResource("/fidelity.txt").getPath()));
+        List<String> al = new ArrayList<String>();
+     /*   String line = null;
+        while ((line = br.readLine()) != null) {
+
+            al.add(line);
+        }
+        br.close(); */
+    	String content = FileUtils.readFileToString(new File("/Users/bgalitsky/Documents/relevance-based-on-parse-trees/fidelity.txt"));
+        String[] als = content.split("\n");
+    	al = Arrays.asList(als);
+    	return al;
+    }
+
+    public void runTest(List<String> lst) throws Exception
+    {
+        System.out.println("Enter text: ");
+        Scanner scanner = new Scanner(System.in);
+        String queryStr = scanner.nextLine();
+        if("quit".equals(queryStr))
+        {
+            return;
+        }
+        else
+        {
+            checkLinguisticScores(queryStr,lst);
+            runTest(lst);
+        }
+
+    }
+
+    public void checkLinguisticScores(String q, List<String> aList) throws Exception
+    {   // convert query into list of tokens
+    	List<String> queryTokens = TextProcessor.fastTokenize(q.toLowerCase(), false);
+    	
+    	List<String> shortListedClasses = new ArrayList<String>();
+    	for (String ans: aList) {
+    		// convert answer class into the list of tokens
+    		List<String> classTokens = TextProcessor.fastTokenize(ans.toLowerCase(), false);
+    		// do intersection of tokens
+    		classTokens.retainAll(queryTokens);
+    		int tokenScore = 0;
+    		// count significant tokens / no stopwords
+    		for(String word: classTokens){
+    			if (word.length()>2 && StringUtils.isAlpha(word))
+    				tokenScore++;
+    		}
+    		if (tokenScore>1)
+    			 shortListedClasses.add(ans);
+    	}
+    	// do it again with lower thresh, if too few results
+    	if (shortListedClasses.size()<5)
+    		for (String ans: aList) {
+        		List<String> classTokens = TextProcessor.fastTokenize(ans.toLowerCase(), false);
+        		classTokens.retainAll(queryTokens);
+        		int tokenScore = 0;
+        		for(String word: classTokens){
+        			if (word.length()>2 && StringUtils.isAlpha(word))
+        				tokenScore++;
+        		}
+        		if (tokenScore>=1)
+        			 shortListedClasses.add(ans);
+        	}
+    	// if no overlap give up of do the full list 
+    	if (shortListedClasses.isEmpty())
+    		shortListedClasses = aList;
+    		
+        ParseTreeChunkListScorer parseTreeChunkListScorer = new ParseTreeChunkListScorer();
+
+        ArrayList<Pair<String,Double>> pairList = new ArrayList<Pair<String,Double>>();
+
+        for (String ans: shortListedClasses) {
+
+            List<List<ParseTreeChunk>> res = m.assessRelevanceCache(q, ans);
+            double score1 = parseTreeChunkListScorer.getParseTreeChunkListScoreAggregPhraseType(res);
+            Pair<String,Double> p = new Pair<String, Double>(ans, score1);
+            pairList.add(p);
+        }
+
+        Collections.sort(pairList, Comparator.comparing(p -> p.getSecond()));
+
+      System.out.println("***** '" + q + "' ******* falls into the following categories: ");
+        for (Pair<String, Double> score: pairList) {
+            System.out.println("        " + score.getFirst() + ": " + score.getSecond());
+        }
+
+
+    }
+}
\ No newline at end of file


[20/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/113TeleHTC_Manual_6_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/113TeleHTC_Manual_6_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/113TeleHTC_Manual_6_EN.txt.txt
new file mode 100644
index 0000000..e99bb25
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/113TeleHTC_Manual_6_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Chapter 6 Exchanging Messages 6.1 Messaging Messaging is a central location where you can find all types of messaging accounts , which include text messages ( SMS ) and e-mail accounts . To select a messaging account Do one of the following : \u2022 Tap Start > All Programs > Messaging . The Account Picker screen then opens and you can select an account . \u2022 While you are in a message list of a certain account , for example SMS / MMS , tap Menu > Go To to switch to other messaging accounts . To configure message account settings \u2022 Tap Menu > Options on the Account Picker screen ; or \u2022 Tap Menu > Tools > Options while you\u2019re in a message list . Account Picker screen 124 Exchanging Messages To automatically add a signature in outgoing messages You can automatically add your name and other information as your signature in outgoing SMS and e-mail messages : Note You need to specify your signature for each type of messaging account . 1. Tap Start > All Programs > Messaging , and then
  tap Menu > Options . 2. Tap Signatures . 3. Select the desired account . 4. Select the Use signature with this account check box . 5. In the text box , enter your signature . 6. To add the signature when you reply to or forward messages , select the Use when replying and forwarding check box , and then tap OK . To filter the Inbox message list When the SMS/MMS or e-mail Inbox on your device is full of messages , you can filter your Inbox to display only the messages that contain a particular sender or subject you are searching for . To filter your Inbox 1. Tap Start > All Programs > Messaging and then tap a messaging account . 2. Enter the sender name or e-mail subject you want to look for . As you type characters , the message list narrows down to the sequence of characters you have entered . When filtering your e-mail Inbox , for example , entering \u201c B \u201d , then \u201c E \u201d narrows the list to only the e-mail messages that contain sender names and e-mail subjects that start with
  \u201c BE . \u201d Exchanging Messages 125 6.2 Text Messages Send short text messages ( SMS ) to other mobile phones . If your message is more than 160 characters , it will be sent as two or more separate messages and you will be billed accordingly . Creating and sending a new text message 1. On the Home screen , slide to the Messages tab . 2. Touch the New Message icon ( ) on the upper right side of the screen . 3. To add recipients , enter their mobile phone numbers in the To field , separating the numbers with a semicolon ( ; ) . You can also tap To or tap Menu > Add Recipient to add mobile phone numbers from Contacts . 4. Enter your message . Tips \u2022 To choose from preset messages , tap Menu > My Text and tap a desired message . \u2022 To check the spelling , tap Menu > Spell Check . 5. When finished , tap Send . Note If you want to know when a text message is received , before sending the message that you are composing , tap Menu > Message Options , then select the Request message del
 ivery notification check box . To always receive a delivery notification , tap Menu > Tools > Options in the text message list , tap SMS / MMS , then select the Request delivery notifications check box . Managing text messages from a contact You can easily read , reply or delete text messages that you received from a certain contact by opening his or her contact details and accessing the Messages tab . 1. On the TouchFLO 3D Home screen , slide to the People tab . 2. Touch All People , and then touch the name of the desired contact . 3. The Contact Details screen then appears . Touch the Messages tab ( ) . 126 Exchanging Messages Lists the exchange of text messages between you and the contact . Enter your reply message here and then touch Send . Touch Menu to forward a text message , delete messages , and more . Managing text messages in the Inbox Text messages that are sent to and received from a contact ( or number ) are grouped as a single thread in your Inbox . Threaded SMS shows
  your messages to and from a contact in a single conversation thread . To open a thread and view the exchange of text messages 1. In the SMS / MMS Inbox , tap a message thread to open it . Exchanging Messages 127 2. You can tap links in the messages to directly open contact details , access a web site , send e-mail , or call the displayed number . \u2022 Tap the sender name to open the Contact Details screen . \u2022 If the sender is not in your contacts , tap the mobile phone number to call the number , reply to the sender , or save it as a new contact . Tap a web address to open the link in Opera Mobile . Enter your reply message here . Tap these navigation keys to select a message in the thread . To reply to a text message At the bottom of the screen , type your reply message and then tap Send . To copy a message in the thread to the SIM card 1. In the SMS / MMS Inbox , tap a message thread to open it . 2. Tap the up or down navigation key at the bottom of the on-screen keyboard to sel
 ect a message in the thread . 3. Tap Menu > Copy to SIM . Notes \u2022 You cannot copy a sent message from the thread to the SIM card . \u2022 Copying text messages from your device to the SIM card results in duplicate messages in the Inbox when your SIM card is in use . 128 Exchanging Messages To delete a message in the thread 1. In the SMS / MMS Inbox , tap a message thread to open it . 2. Tap the up or down navigation key at the bottom of the on-screen keyboard to select a message in the thread . 3. Tap Menu > Delete . To delete all messages in a thread In the SMS / MMS Inbox , tap and hold a message thread , and then tap Delete . Placing a call to the sender of a text message While reading a text message , you can directly call the sender by pressing the TALK/ SEND button . See \u201c Making a call from a text message \u201d in Chapter 2 for details . Exchanging Messages 129 6.3 MMS Sending multimedia messages ( MMS ) to your friends and family is fun and easy . You can include pictures , v
 ideos , and audio clips in a multimedia message . Notes \u2022 Multimedia messaging is a charged service and has to be provisioned on your mobile phone account . Contact your wireless service provider to have MMS provisioned as part of your calling plan . \u2022 Make sure the size of MMS messages that you send is within the limits allowed by your wireless service provider . Changing the MMS settings Check your device to see if it is preconfigured with your wireless service provider\u2019s MMS settings . Add the MMS settings of your wireless service provider if there are no preset MMS settings found on your device . To configure MMS message settings 1. Tap Start > All Programs > Messaging > SMS / MMS . 2. Tap Menu > MMS Options . The Settings screen then appears . 3. In the Preferences tab , choose from the available options according to your needs . 4. Tap the Servers tab and check if your device is preset with MMS server settings . If there are no preset settings , tap New and enter the MMS
  settings which you obtained from your wireless service provider . Note If your device already has preset MMS settings , it is recommended that you do not change these settings . If you change the preset settings , your device may not be able to send and receive MMS messages . 5. Tap Done . Note If you add several MMS message service providers to the Settings screen , you can choose one as your default provider . Select the provider 's name in the list , then tap Set As Default . 130 Exchanging Messages Creating and sending MMS messages You can compose MMS messages in a combination of slides , where each slide can consist of a photo , video or audio clip , and/or text . To compose an MMS message 1. Tap Start > All Programs > Messaging > SMS / MMS . 2. Tap Menu > New > MMS . 3. When the Choose a MMS screen opens , tap a preset template , or tap Custom to open a blank MMS message . Note If you prefer to start composing from a blank MMS message every time , select the Always choose cus
 tom check box . 4. In To , enter the recipient\u2019s phone number or e-mail address directly , or tap To , Cc , or Bcc to choose a phone number or an e-mail address from Contacts . Note You may need to scroll up to see the Cc and Bcc fields . 5. Enter a subject for your message . 6. Tap the Insert icon ( )to insert a photo or video clip . When selecting a photo or video clip , you can : \u2022 Tap the photo or video clip to insert it into the MMS message . \u2022 Tap \u2022 Tap \u2022 Tap to take a photo and insert it into the MMS message . to record MMS video and insert it into the MMS message . to quit selection and return to the MMS message . 7. Enter text and insert audio clips by tapping the respective areas . See \u201c To add text to an MMS message \u201d and \u201c To add an audio clip to an MMS message \u201d for details . 8. To add more slides , tap or tap Menu > Slides > Insert Slide . Repeat steps 5 and 6 to add photos/videos , text , and audio to your slides . 9. Tap Send to send the MMS message
  . Tip You can also send an MMS message from Pictures & Videos , Album , or the Camera program . Exchanging Messages 131 To add text to an MMS message When you tap Insert text here on your MMS message , an empty screen appears where you can enter your text . You can insert the following types of information : Enter your own text Choose from common words or phrases from the My Text list Include a Favorites link Add an emoticon Insert contact information Insert calendar appointment Tip To edit or delete a phrase on the My Text list , tap and hold a string , and then tap Edit or Delete from the shortcut menu . To add a new phrase , tap New . To add an audio clip to an MMS message You can add one audio clip per slide in your MMS message . 1. Tap Insert audio on your MMS message . By default , the My Documents is shown . Tap My Music or another folder that contains audio files . When navigating folders , tap the down arrow ( ) to return to the upper folder . \u2022 Tap to play the audio cli
 p , to pause , or to stop playback . 2. Select an audio file . When selected , you can : \u2022 Tap Select to insert it to the MMS message . 132 Exchanging Messages \u2022 Tap Note to quit selection and return to your MMS message . To record a new audio clip and add it to your MMS message , tap . The Record popup window then opens . Tap Record to start recording , and Stop to end the recording . Tap Play to listen to the recorded audio clip , then tap Add . The new audio clip is then inserted to the MMS message . Viewing and replying to MMS messages To view an MMS message 1. In the SMS / MMS Inbox , tap the MMS message . 2. In the threaded message , tap the MMS icon ( ) , and then use the playback controls , , and to view the MMS message . 3. Tap Contents to see a list of files included in the message . On the Message Contents screen , you can do the following : \u2022 To save a file , select it , and tap Menu > Save . \u2022 To save the contents of a text file to the My Text list , tap Menu > 
 Save into My Text . \u2022 To associate a photo to one of your contacts , tap Menu > Assign to Contact . To reply to an MMS message 1. In the SMS / MMS Inbox , select the mobile phone number or name of the contact who sent you the MMS message . 2. In the threaded message , tap the MMS message to open and view an MMS message . 3. After viewing the MMS message , tap Menu > Reply > via MMS to reply with an MMS message or tap Menu > Reply > via SMS to reply with a text message . Exchanging Messages 133 To block a phone number from sending you MMS messages If you do not want to receive MMS messages from a particular sender , you can add the sender\u2019s phone number to the MMS Blacklist . 1. When you receive a new MMS message , open and view the message . 2. To avoid receiving multimedia messages from this sender next time , tap Menu > Show > Contact Details . 3. Tap Menu > Save to Blacklist . 4. Tap Done . Tip To unblock a phone number and allow the sender to send you MMS messages , remove t
 he number from the Blacklist . Go to the SMS / MMS Inbox , tap Menu > MMS Options > Blacklist tab , tap and hold the phone number in the Blacklist , and then tap Delete . 6.4 E-mail You can set up the following types of e-mail accounts on your device : \u2022 Outlook e-mail that you sync with your computer or the Exchange Server . \u2022 E-mail account that you have from an Internet Service Provider ( ISP ) or other e-mail provider . \u2022 Web-based e-mail accounts such as Yahoo ! � Mail Plus , AOL� , and others . \u2022 Custom domain e-mail accounts . The e-mail domain name is hosted by a different e-mail provider . For more information , search for \u201c custom domain e-mail \u201d in the Help on your device . \u2022 Work account that you access using a VPN connection . 134 Exchanging Messages Synchronizing Outlook e-mail with your computer If you have installed the synchronization software on your computer and created a partnership with your device , then your device is ready to send and receive 
 Outlook e-mail . If you have not yet installed the synchronization software nor created a partnership , do so by following the procedures in Chapter 5. Tip To set up your company e-mail account so that you can access Outlook e-mail messages wirelessly , you must set up your device to synchronize via an over-theair connection with your company\u2019s Exchange Server . For more information about synchronizing with your company e-mail server , see Chapter 7. Setting up an Internet e-mail account Use Windows Mobile\u2019s E-mail Setup Wizard to set up a POP3 or IMAP4 e-mail account on your device if you have an e-mail account from an Internet service provider ( ISP ) or other e-mail service provider , or a web-based account such as Yahoo ! Mail Plus or AOL . You can also add a work account that you access using a VPN server connection as a POP3/IMAP4 account . Note You can also set up an Internet e-mail account from the TouchFLO 3D Mail tab using the simplified Mail Setup . See \u201c Mail \u201d i
 n Chapter 3 for details . 1. Tap Start > All Programs > Messaging > Setup E-mail . 2. Enter your E-mail address and the Password for your e-mail account and select the Save password option . Tap Next . 3. If e-mail provider settings are found preconfigured on your device , the E-mail Setup wizard shows a successful message . Tap Next . If the settings are not found on your device , select the Try to get e-mail settings automatically from the Internet checkbox to find and download e-mail server settings from the Internet and then tap Next . Note You may be charged when your device connects to the Internet and downloads the e-mail server settings . Exchanging Messages 135 4. Enter Your name and the Account display name and then tap Next . Note If the E-mail Setup wizard was unsuccessful in finding and downloading settings from the Internet , select Internet e-mail from the Your e-mail provider list on the next screen . You will be asked to enter e-mail server settings . See \u201c To spe
 cify e-mail server settings \u201d for details . 5. In the Automatic Send/Receive list , choose how frequent you want e-mail messages to be automatically sent and downloaded on your device . Note Tap Review all download settings to select download options , set the e-mail format as HTML or plain text , and more . For more information , see \u201c To customize download and format settings . " 6. Tap Finish . To specify e-mail server settings If Auto Setup is unsuccessful , contact your ISP or e-mail provider for the Incoming mail server and Outgoing mail server settings so you can enter them on your device . Other options that you can select include the following : \u2022 Select the Outgoing server requires authentication check box , if required by your provider . \u2022 If the outgoing e-mail server requires a different user name and password for sending e-mail , clear the Use the same user name and password for sending e-mail check box . You will be prompted to enter this information . \u2022 Tap
  Advanced Server Settings , then select the Require SSL check boxes if your e-mail provider uses an SSL connection for more secured e-mail . From the Network connection list , select the data connection that you use for connecting to the Internet . 136 Exchanging Messages To customize download and format settings Before you tap Finish when setting up your POP3 or IMAP4 e-mail account , you can tap the Review all download settings link at the bottom of the screen to choose download options , message format , and other settings . Options Automatic Send/Receive Download messages Send/receive when I click Send Description Select a time interval for connecting to the Internet automatically to send and receive messages . Set the number of days of messages that you want to be downloaded to your device . By default , messages are immediately delivered when you tap Send . If you prefer to save outgoing messages to the Outbox folder first , clear the check box . ( In this case , you will need
  to manually send messages by tapping Menu > Send/Receive ) . This allows your device to data roam when there is a set time interval for connecting to the Internet automatically . Since this may result in higher connection costs , you may want to leave the check box cleared . Choose whether to delete messages from the mail server when you delete them on your device . Select HTML or Plain Text . Select the e-mail download size . If you have large volumes of e-mail , use a lower size or select to download headers only . Use automatic send/ receive schedule when roaming When deleting messages Message format Message download limit Exchanging Messages 137 Creating and sending an e-mail message 1. On the TouchFLO 3D Home screen , slide to the Mail tab . 2. At the right side of the screen , tap an e-mail account you want to use , or slide your finger up or down until you have selected the desired account . 3. Touch the New E-mail icon ( ) . 4. To add recipients , enter their e-mail address
 es , separating them with a semicolon ( ; ) . You can also tap To if you want to add e-mail addresses stored in Contacts . 5. Enter a subject and compose your message . After setting up e-mail accounts on your device , you can start sending and receiving e-mail messages . Tips \u2022 To choose from preset messages , tap Menu > My Text and tap a desired message . \u2022 To check the spelling , tap Menu > Spell Check . 6. To add an attachment to your message , tap Menu > Insert and tap the item you want to attach : Picture , Voice Note , or File . Select the file or picture you want to attach , or record a voice note . 7. When finished composing the message , tap Send . Tips \u2022 For more information about entering text and symbols , see Chapter 4. \u2022 To set the message priority , tap Menu > Message Options . \u2022 If you are working offline , e-mail messages are moved to the Outbox folder and will be sent the next time you connect . 138 Exchanging Messages Viewing and replying to e-mail mess
 ages To view an e-mail message 1. On the TouchFLO 3D Home screen , slide to the Mail tab , select the e-mail account , and then tap Inbox . 2. Tap an e-mail to open it . To download a complete e-mail By default , long e-mail messages that you receive contains only the first few kilobytes of the message . When you open an e-mail , it displays only the headers and part of the message . You must download the whole e-mail to view the complete content . To download a complete e-mail message , open the e-mail , and then do one of the following : \u2022 Finger-scroll to the end of the message , and tap Get entire message and any attachments ( POP3 e-mail account ) or tap Get the rest of this message ( IMAP4 and Outlook e-mail accounts ) . \u2022 Tap Menu > Download Message . Notes \u2022 The Fetch Mail feature , which is available for POP3 and IMAP4 Internet e-mail accounts , downloads an entire e-mail without the need for you to perform a full synchronization . This limits the download to just the
  e-mail message that you want and helps save data cost . Fetch Mail is also available for Outlook E-mail . See \u201c Instant download through Fetch Mail \u201d in Chapter 7 for details . \u2022 Download may take some time , depending on the speed of your Internet connection , size of the whole e-mail , and whether file attachments are automatically downloaded . Exchanging Messages 139 To download file attachments \u2022 POP3 e-mail account : File attachments are automatically downloaded when you download a complete e-mail . \u2022 IMAP4 and Outlook e-mail accounts : File attachments appear below the subject of an e-mail message . Tapping an attachment opens the attachment if it has been fully downloaded or marks it for download the next time you send and receive e-mail . You can set automatic download of file attachments . See \u201c Customizing e-mail settings \u201d in this chapter for details . To reply to or forward a message 1. On the TouchFLO 3D Home screen , slide to the Mail tab , select the e-
 mail account , and then tap Inbox . 2. Open the message and tap Menu > Reply , Menu > Reply All , or Menu > Forward . 3. Enter your reply message , and then tap Send . Notes \u2022 You can receive and view HTML e-mail messages from any type of e-mail account . The HTML format is retained , without layout shifting or resizing . HTML e-mail support in Outlook Mobile is available only if you are synchronizing your device with Microsoft Exchange Server 2007. \u2022 When replying using an Outlook e-mail account , you can customize ActiveSync to exclude addresses , including your own , from being added to the recipient list . Tap Menu > Options in ActiveSync , select the E-mail item , tap Settings , and then tap Advanced . In the My e-mail addresses text box , enter the e-mail addresses that you want to exclude . 140 Exchanging Messages Directly making calls from an e-mail While reading an e-mail , you can directly place a call to the sender , start a conference call , access contact details , 
 and more . \u2022 Tap the sender photo or name to open the Contact Details screen . \u2022 If the sender is not stored in your contacts , tap the e-mail address to send e-mail or save it as a new contact . \u2022 Tap a recipient name to open his or her contact details . \u2022 Tap a recipient\u2019s e-mail address to send e-mail or save it as a new contact . Tap this Talk button to call the sender . Tap links in the message body to access a web site , send e-mail , or call the displayed number . Tap the To or Cc button or press TALK/SEND to place a call or start a conference call . See Chapter 2 for details . Note If the sender\u2019s name or e-mail address is shown on the e-mail message with a Search button ( ) and you have an Exchange Server e-mail account set up on your device , you can tap this button to search for the sender from the Company Directory . For more information about finding contacts in the Company Directory , see Chapter 7. Exchanging Messages 141 Synchronizing e-mail messages Sync
 hronizing e-mail messages ensures that new e-mail messages are downloaded to the device Inbox folder , e-mail messages in the Outbox folder are sent , and e-mail messages deleted from the server are removed from your device . The manner in which you synchronize e-mail messages depends on the type of e-mail account you have . To automatically synchronize an Outlook e-mail account 1. Connect your device to your computer through USB or Bluetooth . Otherwise , connect through Wi-Fi , or a data connection if you are synchronizing Outlook e-mail with the Exchange Server . For more information , see Chapter 7. 2. Synchronization automatically begins , and your device sends and receives Outlook e-mail . To manually synchronize your Outlook or Internet e-mail account On the TouchFLO 3D Home screen , slide to the Mail tab , select the e-mail account , and then tap Menu > Send/Receive . Customizing e-mail settings To change the download size and format for Outlook e-mail 1. Disconnect your dev
 ice from the computer . 2. Tap Start > ActiveSync . 3. Tap Menu > Options , select E-mail , then tap Settings . 4. On the E-mail Sync Options screen : \u2022 Under Message format , select HTML or Plain Text . HTML is only available when connected to Microsoft Exchange Server 2007 or later . \u2022 Under Download size limit , select the desired e-mail size . 5. Close ActiveSync and reconnect your device to the computer . 142 Exchanging Messages To change the download size and format for Internet e-mail 1. Tap Menu > Options on the Account Picker screen , or tap Menu > Tools > Options while you\u2019re in a message list . 2. Tap your POP3 or IMAP4 Internet e-mail account . 3. Tap Download Size Settings . 4. Under Message format , select HTML or Plain Text . 5. Under Download size limit , select the desired e-mail size . 6. Tap Done . To automatically receive attachments on Outlook e-mail messages 1. Tap Start > ActiveSync . 2. Tap Menu > Options . 3. Tap E-mail > Settings , and then select Inc
 lude file attachments . To automatically receive attachments on IMAP4 e-mail messages 1. Tap Menu > Options on the Account Picker screen , or tap Menu > Tools > Options while you\u2019re in a message list . 2. Tap the name of your IMAP4 e-mail account . 3. Tap Download Size Settings . 4. In Download attachments : \u2022 Select All attachments to always download file attachments ; or \u2022 sSelect a file size limit for auto-downloading attachments . 5. Tap Done . To store attachments on the storage card 1. Tap Start > All Programs > Messaging . 2. Tap Menu > Options > Storage tab . 3. Select the When available , use this storage card to store attachments check box . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/114TeleHTC_Manual_7_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/114TeleHTC_Manual_7_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/114TeleHTC_Manual_7_EN.txt.txt
new file mode 100644
index 0000000..1ffcf92
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/114TeleHTC_Manual_7_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Chapter 7 Working With Company E-mail and Meeting Appointments 7.1 Synchronizing with the Exchange Server To keep up-to-date with your company e-mail messages and meeting schedules while you\u2019re out of the office , you can connect your device to the Internet through Wi-Fi or a data connection and synchronize with your company\u2019s Exchange Server . Setting up an Exchange Server connection Before you can synchronize or access information on the Exchange Server , you need to set up an Exchange Server connection on your device . You need to get the Exchange Server name ( must be Outlook Web Access server name ) , domain name , user name , and the password that you use at work from your network administrator and enter them on your device . Note You can set up an Outlook e-mail account that synchronizes with the Exchange Server from the TouchFLO 3D Mail tab using Mail Setup . See \u201c Mail \u201d in Chapter 3 for details . If you have not synchronized your device with your computer , follow 
 these steps to set up an Exchange Server connection . 1. Tap Start > All Programs > Messaging > Setup E-mail . 2. Enter the E-mail address and Password for the e-mail account and select the Save password option . Tap Next . 144 Working With Company E-mail and Meeting Appointments 3. Clear the Try to get e-mail settings automatically from the Internet option and tap Next . 4. In Your e-mail provider , select Exchange server and tap Next . 5. Tap Next again . 6. Select the Attempt to detect Exchange Server Settings automatically option and tap Next . 7. Enter the Domain name and tap Next . 8. In Server address , enter the Exchange Server address and tap Next . 9. Select the items that you want to sync with the Exchange Server . Tips \u2022 To change synchronization settings of an information item , for example , E-mail , select the item , then tap Settings . \u2022 To change the rules for resolving synchronization conflicts , tap Menu > Advanced . \u2022 If you synchronized e-mail messages wit
 h your computer before , open ActiveSync on your device , then tap Menu > Add Server Source to set up an Exchange Server connection . When prompted to select information types for synchronization , you must first clear the E-mail check box under the Windows PC item before you can select E-mail under Exchange Server . \u2022 To change Exchange Server settings , open ActiveSync on your device , and then tap Menu > Configure Server . 10. Tap Finish . Notes Starting synchronization Before you start synchronizing with the Exchange Server , make sure your device has been set up with a Wi-Fi or data connection to the Internet so that you can synchronize over the air . For more information about connections , see Chapter 8. After you finish setting up an Exchange Server connection , your device automatically starts synchronization . Working With Company E-mail and Meeting Appointments 145 To manually start synchronization , tap Start > Settings , and then touch Sync Data . Note If you connect 
 your device to your office computer via a USB or Bluetooth connection , you can use this connection to the computer to \u201c pass through \u201d to the network and download Outlook e-mail messages and other information to your device . 7.2 Working With Company E-mail Messages Your device gives you instant access to your company e-mail messages and lets you manage your messages easier . Direct Push , Fetch Mail , Remote e-mail search , and e-mail flags are just some of the tools you can use to manage your e-mail messages . Note Some messaging features depend on the Microsoft Exchange Server version used in your company . Check with your network administrator for the availability of these features . Automatic synchronization through Direct Push Direct Push technology ( push e-mail feature ) enables you to receive new e-mail messages on your device as soon as they arrive in your Inbox on the Exchange Server . Items such as contacts , calendar and tasks are also immediately updated onto your
  device when these items have been changed or new entries have been added on the Exchange Server . To make Direct Push work , you need to have a Wi-Fi or data connection on your device . You need to perform a full synchronization between your device and the Exchange Server before Direct Push can be enabled . Requirement The Direct Push feature works for your device only if your company is using Microsoft Exchange Server 2003 Service Pack 2 ( SP2 ) with Exchange ActiveSync or later . 146 Working With Company E-mail and Meeting Appointments To turn on Direct Push in Comm Manager 1. Tap Start > Settings , and then touch Communications . Tip You can also tap Start > All Programs > Comm Manager . 2. On the Comm Manager screen , touch the OFF/ON slider at the right side of the Microsoft Direct Push item . To turn on Direct Push in ActiveSync 1. Tap Start > ActiveSync and then tap Menu > Schedule . 2. Select As items arrive in the Peak times and Off-peak times boxes . Note When Direct Push
  is off , you need to manually retrieve your e-mail messages . Scheduled synchronization If you do not want to use Direct Push , you can set a regular schedule for synchronizing Outlook e-mail and information . 1. In ActiveSync on your device , tap Menu > Schedule . 2. Select from the available options to set the synchronization schedule . Tip You can select a shorter interval in the Peak times box and a longer interval in the Off-peak times box , so that your e-mail messages are synchronized more frequently when you are working . To set the days and hours that make up your peak and off-peak times , tap the peak times link at the bottom of the screen . Instant download through Fetch Mail The Fetch Mail feature downloads an entire e-mail immediately without the need for you to perform a full Send/Receive action . This limits the download to just the e-mail message that you want and helps save data cost . Requirement Fetch Mail works for your device only if your company is using Micro
 soft Exchange Server 2007 or later . 1. Tap Start > All Programs > Messaging > Outlook E-mail . 2. Open an e-mail message . Working With Company E-mail and Meeting Appointments 147 3. By default , only the first few words of the message are shown . To download the whole e-mail , finger-scroll to the end of the message , then tap Get the rest of this message . 4. Wait for the remainder of the message body to download . Notes \u2022 For information about changing e-mail sync options such as setting the download size for e-mail , see " Customizing e-mail settings " in Chapter 6. \u2022 When you receive an e-mail that contains a link to a document such as a PDF or Microsoft Office document located on SharePoint or an internal file server , you can tap the link to view the document on your device . You can view the document only if you have a Microsoft Outlook account that synchronizes with Microsoft Exchange Server 2007 or later . Exchange Server must also be set up to allow access to SharePo
 int document libraries or internal file servers . Searching for e-mail messages on the Exchange Server You can access e-mail messages that are not available on your device by searching your Microsoft Exchange Server mailbox . The search results are downloaded and displayed in a Search Results folder . Requirement Your company must be using Microsoft Exchange Server 2007 or later . 1. Tap Start > All Programs > Messaging > Outlook E-mail . 2. Tap Menu > Tools > Search Server . 3. In the Look for text box , enter the search keyword . 4. Choose the date range of messages to search from . 5. In the Look in list , specify whether to search in the Inbox , Sent Items , or All Folders , and then tap Search . Tip To clear the search results and return to the message list , tap Menu > Clear Results . 148 Working With Company E-mail and Meeting Appointments Flagging your messages Flags serve as a reminder for you to follow-up on important issues or requests contained in e-mail messages . Flagg
 ing messages , which has been a useful feature on desktop Outlook E-mail , can also be done in Outlook Mobile on your device . You can flag received e-mail messages on your device . Requirement Flags are enabled only if e-mail messages are synchronized with Microsoft Exchange Server 2007 or later . Flags are disabled or hidden if e-mail messages are synchronized with earlier versions of Microsoft Exchange Server . To flag or unflag a message 1. Tap Start > All Programs > Messaging > Outlook E-mail . 2. Open a message . 3. Tap Menu > Follow Up and select one of the following options : \u2022 Set Flag Mark the message with a red flag to remind yourself to follow it up . \u2022 Complete Flag Mark the message with a check mark to indicate that the issue or request in the e-mail is already completed . \u2022 Clear Flag Remove the flag to unmark the message . Note E-mail message reminders are displayed on your device if the messages are flagged with reminders and synchronized from the Exchange Ser
 ver . Working With Company E-mail and Meeting Appointments 149 Out-of-office auto-reply Outlook Mobile allows you to retrieve and change your out-of-office status . Just like desktop Outlook , Outlook Mobile automatically sends an auto-reply message when you\u2019re not available . To send out-of-office auto-reply messages 1. Tap Start > All Programs > Messaging > Outlook E-mail . 2. Tap Menu > Tools > Out of Office . 3. In the I am currently list , select Out of the Office . 4. Enter your auto-reply message , and then tap Done . 7.3 Managing Meeting Requests When you schedule and send meeting requests from your device , you can invite attendees to your meeting and check their availability . When you receive a meeting request , you can reply by accepting or declining the request . The meeting request also clearly indicates whether or not there are conflicting or adjacent meetings . Requirement Your company must be using Microsoft Exchange Server 2007 or later . To reply to a meeting re
 quest 1. When you receive a meeting request e-mail , a notification is displayed on your device . Open the e-mail . 2. Tap Accept to reply and accept the meeting request , or tap Menu > Decline if you cannot attend the meeting . 150 Working With Company E-mail and Meeting Appointments Tips \u2022 Before responding , you can check your availability during the time of the requested meeting by tapping View your calendar . \u2022 If the time of the meeting conflicts with your other appointments , a \u201c Scheduling Conflict \u201d status appears on top of the e-mail . 3. Choose whether or not to edit your response e-mail before sending , then tap OK . If you accept the meeting request , it is automatically added as an appointment in Calendar on your device . 7.4 Finding Contacts in the Company Directory In addition to having contacts on your device , you can access contact information from your organization\u2019s Company Directory . By having over-the-air access to the Company Directory , you can ea
 sily send e-mail messages and meeting requests to anyone in your company . Requirement Access to the Company Directory is available only if your organization is running Microsoft Exchange Server 2003 SP2 or higher , and you have completed your first synchronization with the Exchange Server . When browsing Contacts , composing e-mail , or creating a meeting request 1. Synchronize with the Exchange Server . 2. Do any of the following : \u2022 While browsing through your contacts on the Contacts or Phone screen , tap Menu > Company Directory . \u2022 In a new e-mail message , tap the To box ( or tap Menu > Add Recipient ) , and then tap Company Directory on the top of the list . \u2022 When creating a meeting request and selecting required and optional attendees in Calendar , tap Company Directory . Working With Company E-mail and Meeting Appointments 151 3. Enter a partial or full contact name and tap Search . In the search results list , tap a contact to select it . 4. You can save a contact 
 from the Company Directory to your device by selecting the contact , and then tapping Menu > Save to Contacts . Note You can search for the following information as long as that information is included in the Company Directory : First name , Last name , E-mail name , Display name , E-mail address , or Office location . ? When viewing a received Outlook e-mail message 1. Open a received Outlook e-mail message . 2. If a Search button ( ) appears to the right of the sender name in the e-mail message , tap this button to search for the sender in the Company Directory . 3. If the sender is found , the contact details will be displayed . You can then choose to save the sender to your contacts , call the sender , and more . 152 Working With Company E-mail and Meeting Appointments 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/115TeleHTC_Manual_8_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/115TeleHTC_Manual_8_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/115TeleHTC_Manual_8_EN.txt.txt
new file mode 100644
index 0000000..05e7f40
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/115TeleHTC_Manual_8_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Chapter 8 Internet 8.1 Ways of Connecting to the Internet Your device\u2019s networking capabilities allow you to access the Internet or your corporate network through Wi-Fi , GPRS/3G ( or EDGE when available ) , or dial-up . You can also add and set up a VPN or proxy connection . Wi-Fi Note Wi-Fi provides wireless Internet access over distances of up to 100 meters ( 300 feet ) . To use Wi-Fi on your device , you need access to a wireless access point or \u201c hotspot \u201d . The availability and range of your device\u2019s Wi-Fi signal depends on the number , infrastructure , and other objects through which the signal passes . To turn Wi-Fi on and connect to a Wi-Fi network 1. Tap Start > Settings , and then tap Communications . 2. Tap Wi-Fi . Detected wireless networks are displayed on the screen . 3. Tap the desired Wi-Fi network . \u2022 If you selected an open ( unsecured ) network , you are automatically connected to the network . \u2022 If you selected a secured network , enter the security 
 key and then tap Done . You are then connected to the network . 154 Internet Indicates that your device is connected to this Wi-Fi network Indicates a secured Wi-Fi network . Signal strength Network name ( SSID ) 4. Tap Back to return to the Communications screen . Next time you use your device to detect Wi-Fi networks , you will not be prompted to enter the network key of the previously accessed Wi-Fi network ( unless you perform a hard reset which will erase custom settings on your device ) . Notes \u2022 Wi-Fi networks are self-discoverable , which means no additional steps are required for your device to connect to a Wi-Fi network . It may be necessary to provide a username and password for certain closed wireless networks . \u2022 To turn off Wi-Fi , on the Communications screen , tap the OFF/ON slider at the right side of the Wi-Fi item . Internet 155 To check Wi-Fi network status Aside from the Wi-Fi status icon on the title bar , you can check the current Wi-Fi connection status f
 rom any of the following screens of your device : \u2022 Wireless LAN screen Tap Advanced on the Wi-Fi screen . On the Main tab , you will see the Wi-Fi network that your device is currently connected to . The configuration and signal quality of the Wi-Fi network are also shown . \u2022 Configure Wireless Networks screen Tap Start > Settings > All Settings > Connections tab > Wi-Fi and then tap the Wireless Networks button . This screen displays all available Wi-Fi networks in range . To connect to a Wi-Fi network in the list , tap and hold on the desired network , and then tap Connect . Tap a Wi-Fi network in the list to view or change its connection settings . You can also add new Wi-Fi networks , if available , by tapping Add New . 156 Internet To save battery power while connected to a Wi-Fi network Using Wi-Fi quickly consumes battery power . Turn off Wi-Fi when not in use . You can also enable power saving settings while using Wi-Fi . 1. Tap Start > Settings , and then tap Menu > Wi
 reless Networks > Power Mode tab . 2. Move the Power Save Mode slider to a position that optimizes performance with the least power consumption . For example , move the slider to the left ( Best Performance ) to have the optimal wireless network performance ; move to the right ( Best Battery ) to obtain the maximum battery usage . GPRS/3G Use GPRS/3G ( or EDGE , if available ) to connect to the Internet and to send and receive email on your device . You will need a data plan to use your wireless service provider\u2019s GPRS/3G network . Check with your wireless service provider to find out about GPRS/3G rates . If you completed the Connection Setup process for auto-configuring the data connection the first time you turned on your device , your device is ready to use your wireless service provider\u2019s GPRS/3G services . Do not change the connection settings as this may cause services to stop working . If you need to add another GPRS/3G connection on your device , contact your wireless s
 ervice provider . Note If you did not complete the auto-configuration process before , follow the instructions in \u201c To manually select the network operator to use for data connections \u201d in Chapter 13 to configure the data connection for your device . Internet 157 Dial-up To set up a dial-up connection to your Internet Service Provider ( ISP ) on your device , you need the same settings that you normally use when you dial up from your computer . This includes the ISP server phone number , your user name and password . 1. Tap Start > Settings , and then tap All Settings . 2. On the Connections tab , tap Connections . 3. Tap Add a new modem connection . 4. In the Select a modem list , select Cellular Line , then tap Next . 5. Enter the ISP server phone number , then tap Next . 6. Enter your user name , password , and any other information required by your ISP , then tap Finish . Starting a data connection After setting up a data connection such as GPRS/3G or ISP dial-up on your dev
 ice , you can now connect your device to the Internet . The connection is automatically started when you begin using a program that accesses the Internet such as Opera Mobile . To manually start a data connection If you have set up multiple types of data connections on your device , you can manually start a connection . 1. Tap Start > Settings , and then tap All Settings . 2. On the Connections tab , tap Connections . 3. Tap Manage existing connections . 4. Tap and hold the name of the desired connection , and then tap Connect . To disconnect active data connections 1. On the Home screen , slide to the Settings tab , and then tap Communications . 2. Tap the OFF/ON slider at the right side of the Data Connection item . 158 Internet 8.2 Using Opera Mobile\u2122 Opera Mobile\u2122 is a full-featured Internet browser , optimized for use on your device . To open Opera Mobile and go to a website 1. On the Home screen , slide to the Internet tab and then tap Launch Browser . 2. On the Opera Mobi
 le screen , tap the address bar . Note If the address bar is hidden , tap at the bottom-right corner of the screen . 3. Enter the URL address and tap . The browser screen 1 8 2 3 1 2 4 5 6 7 9 Refresh the web page being viewed . Enter the URL of the web site here . Internet 159 3 4 5 6 7 8 9 Return to the previous web page viewed . Go to the Bookmarks screen where you can add , edit , or send bookmarks . Add a new browsing tab or switch to another browsing tab . Go to the home page . Open a list where you can change Opera Mobile settings . See \u201c Opera Mobile menu \u201d in this chapter for details . Close the current tab . This is enabled only if you have more than one tab opened . When browsing the web , Opera Mobile automatically switches to full screen mode . Tap to show the address and Menu bars again . Opera Mobile menu On the menu bar , tap to open the menu . Open the Settings screen where you can set your home page , change display settings , set privacy settings , and more . 
 View items being downloaded , and resume or stop current downloads . View information about the web page you are browsing . See web pages that you have viewed . Manage and view web pages that you have saved . Open the Opera Mobile help file . Close Opera Mobile completely . Note For more information about using Opera Mobile , go to : http://www.opera.com/products/mobile/. 160 Internet Navigating web pages To pan on the web page While browsing web pages , use finger gestures to pan on the web page . Tip Touch and hold an empty area on a web page to open a shortcut menu that lets you go back to the previous page , reload the page , add the page as a Favorite , and more . You can also tap and hold a link or image on a web page to open a menu with different options . To zoom in and out on web pages While viewing a web page in full screen , slide to the right of the zoom bar to zoom in on the web page . To zoom out , slide your finger to the left . Tip You can also tap the screen twice t
 o zoom in . To zoom out , tap the screen twice again . Zooming in Zooming out Internet 161 To change the screen orientation Change the screen orientation by simply turning your device sideways . 162 Internet 8.3 Using YouTube\u2122 YouTube\u2122 lets you browse , search , and watch Internet videos that are posted in YouTube . Note You need an active data or Wi-Fi connection to use YouTube . If you are using a GPRS/3G connection , watching Internet videos may significantly increase your data connection costs . To open YouTube On the Home screen , slide to the Internet tab , and then tap YouTube . Open the Program Guide . Displays the current program channel being viewed . Set the quality of downloaded videos and site ( country ) preference . Close YouTube . Filter the list by type of video ( Featured , Most Viewed , Top Favorited , or Top Rated ) and by the time period when videos were uploaded to the YouTube server . Displays the available videos for the selected program channel . Tap a v
 ideo to watch it . Tap to view details about the video . Switch between the YouTube screens . Searching videos 1. On the YouTube screen , tap . 2. On the Search screen , enter the video information , and then tap Search . Internet 163 YouTube searches for the video , and then displays all related videos on the screen . 3. Tap on a video to watch it . See \u201c Watching videos \u201d for details . Using the Program Guide The Program Guide categorizes videos into channels so you can browse and watch related videos available on YouTube . You can also add your favorite video channel in the Program Guide . To add a new channel 1. Search the channel or video you want to add to the Program Guide . See \u201c Searching videos \u201d for details . 2. In the search results list , tap at the right side of the video or channel to open the details screen . 3. Tap , and then tap OK on the confirmation screen . The channel you just subscribed to is shown when you tap Program Guide . Tip You can also add a se
 arch information as a program channel . To change the program channel 1. On the Videos screen , tap Program Guide at the top of the screen . 2. On the Program Guide , select a channel . 164 Internet Tip You can scroll down to view more channels . The video list refreshes to display the available videos for the selected channel . To delete a program channel You can only delete channels that you have added to the Program Guide . 1. Tap Program Guide . 2. Scroll down to the channel that you want to delete , and then tap 3. On the confirmation screen , tap Yes . . Watching videos Tap a video on any YouTube screen , program guide channel , or search results list to launch the YouTube Video Player . Drag to go forward or backward in the video . Download progress Go back to the beginning of the video . Buffered video Depending on the quality of downloaded videos you selected , tap to view the video in its actual size , in expanded size , or in full screen . Play/pause the video . Stop the 
 video playback . Tip To adjust the volume , press the VOLUME UP and VOLUME DOWN buttons on the left panel of the device . Internet 165 When you tap Stop ( the following options . ) or finish watching the entire video , you can select from Go back to the previous page of related videos . Send the link of the previously watched video via e-mail , or to send the video thumbnail via MMS . Go to the next page of related videos . If no action is taken , the next page is automatically displayed after 10 seconds . View information about the video you just watched . Add the video to your Bookmarks . Replay the video . Bookmarks You can save your favorite videos in Bookmarks for quick access . To save a video in Bookmarks After watching a video , tap . You can also tap on any video that you want to add to Bookmarks and then tap on the video details screen . To watch videos in Bookmarks On the YouTube screen , tap . Tap a video to watch it . 166 Internet To delete a video from Bookmarks 1. On 
 the Bookmarks screen , tap Edit . 2. Tap on the video you want to remove . 3. On the confirmation screen , tap Yes . 4. Tap Done . History Note View recently watched videos on the History screen . On the YouTube screen , tap and then tap a video to watch it . To clear the History list , tap Clear . 8.4 Using Streaming Media Streaming Media lets you stream live broadcasts or on-demand media content from the web . The media is sent in a continuous stream to your device and is played as it arrives , along with the audio . The Streaming Media program allows you to play 3GP and MPEG-4 files . It also supports playback of SDP ( Session Description Protocol ) files . To access streaming content via Opera Mobile In Opera Mobile , you can open a web page containing RTSP ( Real Time Streaming Protocol ) links to streaming media content then play the content in the Streaming Media program . 1. On the Home screen , slide to the Internet tab and tap Launch Browser . 2. In the address bar , enter
  the URL address of the web page that contains the RTSP link to the desired *.3gp , * . mp4 , or * . sdp file . 3. On the web page , tap the RTSP link . Internet 167 4. Streaming Media then automatically opens and starts playing the file . During playback , use the slider and button controls to play/pause , switch to full screen , replay , and fast forward . Tap to toggle between playing or pausing the video playback . Play video from the beginning . Switch to full screen mode . In full screen mode , tap to switch back . Drag the slider to go backward or forward on the video . View the video in its actual size , in expanded size , or in full screen . Stop playback . To stream media files directly on the Streaming Media Player Streaming Media only supports RTSP links when you stream *.3gp and * . mp4 files directly on the program . 1. Tap Start > All Programs > Streaming Media . 2. Tap Menu > Connect . 3. Enter the complete RTSP link , and then tap Connect . Streaming Media starts pl
 aying the file . 4. During playback , use the slider and playback control buttons to play/pause , switch to full screen , replay , and fast forward . 168 Internet To view streaming video information While streaming a video clip , tap Menu > Properties to view the video properties such as the video link or location , duration , and bit rate . To configure streaming video connection settings Tap Menu > Options to configure the Streaming Media settings . From this screen , you can set the media buffer , select the connection to use , and set all streaming videos to play in full screen . 8.5 Using Windows Live\u2122 Windows Live\u2122 helps you quickly find information using Windows Live\u2122 Search , access and synchronize your Windows Live\u2122 Mail e-mails and contacts , and send instant messages using Windows Live\u2122 Messenger . To set up Windows Live\u2122 for the first time 1. Tap Start > All Programs > Windows Live , and then tap Sign in to Windows Live . 2. Tap the links to read the Windows 
 Live\u2122 Terms of Use and the Microsoft Online Privacy Statement . After reviewing them , tap Accept . 3. Enter your Windows Live\u2122 Mail or Hotmail address and password , select the Save password checkbox , and then tap Next . 4. Tap Next . 5. Choose what information you want to sync online with your device . If you select Store Windows Live contacts in your mobile phone\u2019s contact list , your Windows Live\u2122 contacts will be added to the contact list and to Windows Live\u2122 Messenger on your device . Internet 169 If you select Sync e-mail , your Windows Live\u2122 Mail or Hotmail inbox messages will be downloaded to your device . 6. Tap Next . 7. After synchronization is complete , tap Done . The Windows Live\u2122 screen Search bar . Enter the information you want to search , then tap . Switch between the Windows Live\u2122 Messenger , Windows Live\u2122 Mail , and Sync status screens . Change your display name , picture , and personal message . Adjust Windows Live\u2122 settings . Using Windows
  Live\u2122 Messenger With this mobile version of Windows Live\u2122 Messenger , you can send and receive instant messages . To sign in to Windows Live\u2122 Messenger and start a conversation 1. Tap Messenger on the Windows Live\u2122 screen . 2. Tap Sign in on the Messenger screen . Signing in may take several minutes , depending on your connection speed . 170 Internet 3. In the contact list , select an online contact , and then tap Send IM to open the message screen . 4. On the message screen , enter your message in the text field , then tap the Enter key on the on-screen keyboard to send the message . You can also do any of the following on the message screen : \u2022 To add an emoticon , tap Menu > Add emoticon , then tap an emoticon on the screen . The emoticon is attached to your message . To send a file , tap Menu > Send , select the type of file ( picture , voice clip , or any file ) , and then locate the file you want to send . To invite one or more contacts to the conversation , tap Men
 u > Options > Add participant . To send a voice message , tap Voice clip on the message screen , then start talking . When you have finished recording , tap Send . \u2022 \u2022 \u2022 5. To end the conversation with the current contact , tap Menu > End conversation . 6. To sign out , tap Menu > Sign Out . To add a Windows Live\u2122 contact You can add new Windows Live\u2122 contacts in Windows Live\u2122 Messenger . 1. Tap Menu > Add new contact . 2. Enter the e-mail address of the contact , and then tap OK . Internet 171 8.6 Using RSS Hub RSS Hub is an RSS reader that keeps you up-to-date with the latest news from the Internet . It monitors RSS feeds from your favorite news sites , blogs and more , and downloads the most current updates and podcasts . For more information about RSS Hub , tap Start > All Programs > Help , and then tap Help for Added Programs > RSS Hub . Subscribing to and organizing news channels To subscribe to news channels 1. Tap Start > All Programs > RSS Hub , and then tap Con
 tinue . 2. In the New Channel Wizard screen , select the Choose from a list of known Channels option , and then tap Next . A list of RSS channels is displayed . Note If the RSS channel you want to subscribe to is not in the list of known channels , tap Back , then select another option on the New Channel Wizard screen . You must subscribe to news channels before you can receive RSS feeds . Tap to load pre-selected RSS channels . 172 Internet 3. Select the checkboxes of the RSS channels you want to subscribe to , then tap Finish . Category Channels Tip Channels are grouped by categories . If you want to select all channels in a category , select the category check box . 4. The channel list appears . Tap Refresh to update the channels . Internet 173 Select the channel category to display Channel is updated . Tap to view the headlines . Channel list Headline count Update the channels Cancel channel updating To add new channels 1. In the channel list , tap Menu > Channel > New to open t
 he New Channel Wizard screen . 2. Select an option , and then follow screen instructions to add new channels . To filter the channels to display In the channel list , tap Menu > View , and then select Updated to display only those channels with unread items , or All to display all subscribed channels . 174 Internet Using the headline list Tap a channel in the channel list to display the headline list . Tap a headline to open the news summary screen . Play back the podcast or view the image . New or unread item Item has been read Podcast or image attachment is downloading . Download the podcast or image attachment . The item contains a keyword . Return to the channel list New or unread items display in bold and are marked with a blue newspaper icon . Items that have been read are marked with a gray newspaper icon . Internet 175 Viewing the news summary Tap a headline in the headline list to display the news summary . Channel name Headline and read status News summary Open the complet
 e article in Opera Mobile . See \u201c Using Opera Mobile \u201d in this chapter for details . Return to the headline list Podcast or image attachment . Tap the link to download the file . Tap Menu to change the way that text is displayed , save a podcast , send a link to the article via e-mail , and more . To download , play , or save podcasts 1. On the news summary screen tap the Download audio/Download video link or the audio/video podcast icon . Tip To cancel the file download , tap Cancel download below the audio/video file icon . 2. After downloading the file , do one of the following : \u2022 Tap Play audio/Play video or the audio/video file icon to play back the file . \u2022 Tap Menu > Podcast > Save to save a copy of the file to the device\u2019s main memory or storage card . 176 Internet To send the a link to the article via e-mail 1. In the news summary screen , tap Menu > Send Link via Email . 2. Messaging opens and displays the Account Picker screen . Tap the e-mail account that you 
 want to use . A new e-mail message is then created with the link . 3. Specify the e-mail recipients , and then tap Send . Note For more information on how to compose e-mails , see \u201c Creating and sending an e-mail message \u201d in Chapter 6. To view the next or previous news summary In the news summary screen , tap Menu > Next or Menu > Previous . 8.7 Using Your Device as a Modem ( Internet Sharing ) Internet Sharing connects your desktop or notebook computer to the Internet by using your device\u2019s data connection such as GPRS/3G . You can connect via USB or Bluetooth . Before you proceed \u2022 Make sure your device has a SIM card installed , and your device has a GPRS/3G or dial-up modem connection . If your device has not been set up with a data connection yet , tap Menu > Connection Settings on the Internet Sharing screen . For more information about setting up these connections , see \u201c GPRS/3G \u201d and \u201c Dial-up \u201d in this chapter . To set up your device as a USB modem , you m
 ust first install Windows Mobile Device Center or Microsoft ActiveSync version 4.5 or later on the computer . To use a Bluetooth connection , make sure your computer has built-in Bluetooth capability . If your computer does not have Bluetooth , you can connect and use a Bluetooth adapter or dongle . \u2022 \u2022 Internet 177 Setting up your device as a USB modem 1. Connect the device to your desktop or notebook computer using the USB sync cable . 2. When the Connect to PC screen appears , select Internet Sharing , then tap Done . 3. In the Network Connection list , select the name of the connection that your device uses to connect to the Internet . 4. Tap Connect . Using your device as a Bluetooth modem To use your device as a Bluetooth modem for your computer , set up a Bluetooth Personal Area Network ( PAN ) between your device and your computer . To set up Bluetooth PAN on your device 1. On your device , turn on Bluetooth and set it to visible mode . For instructions , see \u201c To turn
  Bluetooth on and make your device visible \u201d in Chapter 9. 2. Initiate a Bluetooth partnership from your device . For instructions , see \u201c To create a Bluetooth partnership \u201d in Chapter 9. 3. Tap Start > All Programs > Internet Sharing . 4. Select Bluetooth PAN as the PC Connection . 5. From the Network Connection list , select the name of the connection that your device uses to connect to the Internet , and then tap Connect . 178 Internet 6. Set up Bluetooth PAN on your computer . For instructions , see \u201c To set up Bluetooth PAN on your computer \u201d . 7. On the Internet Sharing screen on your device , check whether a connected status message is displayed , which indicates that your computer has been successfully connected to the Internet using your device as a Bluetooth modem . To set up Bluetooth PAN on your computer For Windows Vista : 1. Click Start > Control Panel > Network and Internet > Network and Sharing Center . 2. Click Manage network connections and then under Pe
 rsonal Area Network , double-click Bluetooth Network Connection . 3. In the Bluetooth Personal Area Network Devices dialog box , select your device , and click Connect . For Windows XP : 1. Tap Start > Control Panel > Network Connections . 2. Under Personal Area Network , click the Bluetooth Network Connection icon . 3. Under Network Tasks , click View Bluetooth network devices . 4. In the Bluetooth Personal Area Network Devices dialog box , select your device , and click Connect . Note If the Bluetooth adapter or dongle on your computer was installed using a third-party driver , open the Bluetooth software that came with the Bluetooth adapter/dongle to connect your computer and device via Bluetooth . Refer to the Bluetooth adapter/dongle documentation for more information . Ending the Internet connection On the Internet Sharing screen , tap Disconnect . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/116TeleHTC_Manual_9_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/116TeleHTC_Manual_9_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/116TeleHTC_Manual_9_EN.txt.txt
new file mode 100644
index 0000000..6e7be31
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/116TeleHTC_Manual_9_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Chapter 9 Bluetooth 9.1 Bluetooth Modes Bluetooth is a short-range wireless communications technology . Devices with Bluetooth capabilities can exchange information over a distance of about eight meters ( 26 feet ) without requiring a physical connection . Bluetooth on your device operates in three different modes : \u2022 On . Bluetooth is turned on . Your device can detect other Bluetooth-enabled devices , but not vice versa . \u2022 Visible . Bluetooth is turned on , and all other Bluetooth-enabled devices can detect your device . \u2022 Off . Bluetooth is turned off . In this mode , you can neither send nor receive information using Bluetooth . Note Turn off Bluetooth when not in use to conserve battery power , or in places where using a wireless device is prohibited , such as on board an aircraft and in hospitals . To turn Bluetooth on and make your device visible 1. Tap Start > Settings > Communications and then tap Bluetooth . Tip You can also tap Start > Settings > All Settings > Con
 nections tab and then tap Bluetooth . 2. When prompted to turn on Bluetooth , tap Yes . 3. Tap the Settings tab ( ) , and then tap Not Discoverable . 180 Bluetooth 4. Tap the drop-down menu and then select Always Visible . Note If you select Visible for 60 seconds , Bluetooth visibility will turn off automatically when the time elapses and your device has not yet connected to another Bluetooth device . 5. Tap Done . The icon then appears in the title bar to indicate that the device is in visible mode . Tip To easily switch Bluetooth off or back on , tap Start > Settings > Communications to open the Comm Manager and then tap Bluetooth\u2019s OFF/ON slider . Your device remembers whether visible mode was previously enabled or disabled . If you did not disable visible mode , switching Bluetooth on in the Comm Manager also puts your device in visible mode automatically . For more information about using the Comm Manager , see Chapter 13. 9.2 Bluetooth Partnerships A Bluetooth partnership i
 s a relationship that you create between your device and another Bluetooth-enabled device in order to exchange information in a secure manner . To create a Bluetooth partnership 1. Tap Start > Settings > Communications and then tap Bluetooth . 2. On the Settings tab ( ) , tap Add a device . 3. Tap a Bluetooth device to connect with . 4. Specify a passcode , which can be 1 up to 16 characters , to establish a secure connection , and then tap OK . Note Your device uses Bluetooth 2.1 with Secure Simple Pairing feature . If the other Bluetooth device you are pairing with also supports the same feature , the passcode will be automatically created and displayed on both devices . Both you and other party simply need to verify that the passcode is the same and tap Yes to establish the partnership . Bluetooth 181 5. After the receiving party accepts the partnership , the Bluetooth device name is shown on the screen , including all Bluetooth services that you can use with it . Select the chec
 k boxes of the Bluetooth services you want to use , and then tap Done . 6. If you selected to use the Bluetooth Serial Port service of the paired device , select a local serial port to assign for the paired device , and then tap Continue . The connected Bluetooth device now appears in the Devices tab ( Tip Tap Properties to change the paired device name . Tap Menu > Update to update the list of services that you can use with the paired device . ) . Creating a Bluetooth partnership between two devices is a one-time process . Once a partnership is created , the devices can recognize the partnership and exchange information without entering a passcode again . To accept a Bluetooth partnership 1. Ensure that Bluetooth is turned on and in visible mode . 2. When prompted , enter or accept the passcode ( the same passcode from the device requesting the partnership ) to establish a secure connection . 3. Tap OK . The paired device now appears in the Devices tab ( now exchange information wi
 th the paired device . Tip ) . You can To delete a Bluetooth partnership , select the device\u2019s name on the Devices tab , then tap Menu > Delete . When prompted to confirm , tap Yes . 182 Bluetooth 9.3 Connecting a Bluetooth Hands-free or Stereo Headset For hands-free phone conversations , you can use a Bluetooth hands-free headset such as a car kit with your device . Your device also supports A2DP ( Advanced Audio Distribution Profile ) which allows you to use a Bluetooth stereo headset for handsfree phone conversations and for listening to stereo music . To connect a Bluetooth hands-free or stereo headset 1. Make sure that both your device and the Bluetooth headset are turned on and within close range , and that the headset is visible . Refer to the manufacturer\u2019s documentation to find out how to set the headset in visible mode . 2. Tap Start > Settings > Communications and then tap Bluetooth . 3. On the Settings tab ( ) , tap Add a device . 4. Select the hands-free or stereo h
 eadset device in the list , then tap Select . Your device will automatically with the Bluetooth headset . Note Your device will automatically try one of the preconfigured passcodes ( 0000 , 8888 ) to pair with the Bluetooth headset . If this fails , you have to manually enter the passcode delivered with your headset . 5. When connected , the hands-free or stereo headset device is shown on the screen , including the Bluetooth services ( Hands-free and Stereo Audio ) that you can use with it . Make sure these services are selected , and then tap Continue . The paired headset now appears in the Devices tab ( ) . Bluetooth 183 The double arrowhead that is displayed to the right of a service indicates that the service is connected with your headset . If you turn off your headset , the double arrowhead disappears to indicate that the service has been disconnected . Next time you turn on and use your headset again , you need to manually reconnect each service . Tap a service and then tap C
 onnect . 9.4 Sending and Receiving Information Using Bluetooth You can send the following types of information from your device to your computer or to another Bluetooth-enabled device : Outlook e-mail , contacts , tasks , notes , calendar items , audio files , ring tones , Images , and videos . Note If your computer does not have built-in Bluetooth capability , you need to connect and use a Bluetooth adapter or dongle on your computer . To send information from your device to a computer 1. Turn on Bluetooth on your computer and set to visible mode . Note If the Bluetooth adapter or dongle on your computer was installed using a thirdparty provided driver , open the Bluetooth software that came with the Bluetooth adapter/dongle and enable the adapter/dongle to be discovered by other devices . Refer to the Bluetooth adapter\u2019s documentation for more information . 184 Bluetooth If your computer is installed with Windows XP SP2 or later version and your computer\u2019s Bluetooth adapter is
  supported by your Windows version , do the following steps : a. On your computer , open Bluetooth Devices from the Control Panel , then click the Options tab on the Bluetooth Devices window . b. For Windows Vista , select Allow Bluetooth devices to find this computer . For Windows XP , select Turn discovery on and Allow Bluetooth devices to connect to this computer . c. Create a Bluetooth partnership between your device and computer . For instructions , see \u201c To create a Bluetooth partnership \u201d . d. In the Options tab of Bluetooth Devices , select Show the Bluetooth icon in the notification area . e. To enable your computer to receive information via Bluetooth , right-click the Bluetooth icon in the notification area at the bottom-right corner of your computer screen and select Receive a File . You are now ready to send/receive . 2. Turn on Bluetooth on your device and make your device visible . For instructions , see \u201c To turn Bluetooth on and make your device visible \u201d . 
 3. Create a Bluetooth partnership . For instructions , see \u201c To create a Bluetooth partnership \u201d . 4. Tap the Accessories tab ( then tap BT Send Object . ) , and Tap the left or right arrow to change to the type of information you want to send . 5. Select the type of information and then select the check boxes of items you want to send . 6. Tap Send . 7. Select the Bluetooth device to which you want to send the items , then tap Select . Bluetooth 185 8. If you sent Outlook E-mail messages , calendar items , tasks , or contacts to your computer and they are not automatically added to Outlook , select File > Import and Export in Outlook to import them . To send information to a Bluetooth-enabled device such as another Windows Mobile-powered device , follow steps 2 to 7 in the above procedure . Tip The default folder on your computer where sent items are stored may be C : \Documents and Settings\your_username\My Documents\Bluetooth Exchange Folder in Windows XP or C : \Users\your_u
 sername\My Documents\Bluetooth Exchange Folder in Windows Vista . To send a file to another Bluetooth-enabled device 1. Tap Start > All Programs > File Explorer . 2. Navigate to the folder that contains the file you want to send . 3. Tap and hold the desired file , and then tap Send via Bluetooth . 4. Select the name of the Bluetooth device , and then tap Select . To receive information from another Bluetooth-enabled device 1. When you receive information such as calendar items , tasks , or contacts sent from another Bluetooth-enabled device , tap Accept . 2. After your device has finished saving the information , tap Done . To change the folder for saving received files Files that you receive from another Bluetooth-enabled device are saved in \My Documents on your device . You can change the destination folder for saving received files : 1. Tap Start > Settings > Communications and then tap Bluetooth . 2. On the Settings tab ( 4. Tap Search ( ) . ) , tap Advanced . 3. Scroll down a
 nd then tap File Transfer . 186 Bluetooth 5. Select the new destination folder , and then tap Select . Tip To save received files on your storage card when it is inserted on your device , select Save files to storage card when present . Files will be saved on the root folder of your storage card . 6. Tap Done . 9.5 Printing via Bluetooth Connect your device to a Bluetooth printer to print your pictures . Note Before you start printing , make sure to turn on the Bluetooth printer and turn on Bluetooth on your device . For instructions , see \u201c To turn Bluetooth on and make your device visible \u201d earlier in this chapter . To print a picture 1. Tap Start > Settings > Communications and then tap Bluetooth . 2. Tap the Accessories tab ( ) , and then tap Print Images . Your device then searches for pictures that are stored on your device and storage card . 3. Tap each picture that you want to select for printing . To select all pictures , tap Menu > Select All . Tip To clear a selected 
 picture , tap that picture again . 4. After you have finished selecting pictures , tap Next . 5. Select the name of the Bluetooth printer , and then tap Select . 6. Select the Print Layout to use , and then tap Print . Bluetooth 187 9.6 Connecting to Car Kit Phones Bluetooth car kit phones that support the SIM Access Profile ( SAP ) can connect to your device and access its SIM card via Bluetooth . Once a SAP connection has been established , you can download device and/or SIM contacts from your device to the car kit phone , and use the car kit phone to answer and make calls . 1. Tap Start > Settings > Communications and then tap Bluetooth . 2. Turn Bluetooth on and set your device to visible mode . For instructions , see \u201c To turn Bluetooth on and make your device visible \u201d earlier in this chapter . 3. On the Settings tab ( ) , tap Advanced . 4. Scroll down and tap SIM Access . 5. Select the Enable SIM Access Profile ( SAP ) check box , and then tap Done . 6. Connect the car ki
 t phone to your device via Bluetooth . For instructions , refer to your car kit phone\u2019s manual . 7. If this is the first time you are pairing the car kit phone with your device , enter the car kit phone\u2019s passcode on your device . 8. A pop-up message should appear on your device saying that a connection has been established between your device and car kit phone . Tap OK . Note You are not able to make and receive calls , send and receive messages , or perform related activities on your device when the SIM Access Profile connection is active . You\u2019ll be able to make or receive calls from the car kit phone during this time . After the connection is disconnected , you will be able to use your device as before . 188 Bluetooth 
\ No newline at end of file


[46/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/NERPhraseGeneralizer.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/NERPhraseGeneralizer.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/NERPhraseGeneralizer.java
new file mode 100644
index 0000000..a5e1ee7
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/NERPhraseGeneralizer.java
@@ -0,0 +1,271 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.parse_thicket.matching;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+
+
+
+
+
+
+
+
+
+
+import opennlp.tools.parse_thicket.IGeneralizer;
+import opennlp.tools.parse_thicket.ParseTreeNode;
+import opennlp.tools.stemmer.PStemmer;
+import opennlp.tools.textsimilarity.GeneralizationListReducer;
+import opennlp.tools.textsimilarity.LemmaFormManager;
+import opennlp.tools.textsimilarity.POSManager;
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+
+public class NERPhraseGeneralizer extends PhraseGeneralizer {
+
+	/* alignment is based on NER values, not on POS now
+	 * 
+	 */
+
+
+	/**
+	 * key matching function which takes two phrases, aligns them and finds a set
+	 * of maximum common sub-phrase
+	 * 
+	 * @param chunk1
+	 * @param chunk2
+	 * @return
+	 */
+	@Override
+	public List<ParseTreeChunk> generalize(
+			Object chunk1o, Object chunk2o) {
+
+		ParseTreeChunk chunk1 = (ParseTreeChunk)chunk1o, chunk2 = (ParseTreeChunk)chunk2o;
+		List<ParseTreeNode> results = new ArrayList<ParseTreeNode>();
+		List<ParseTreeChunk> resultChunks = new ArrayList<ParseTreeChunk>();
+
+
+		List<String> pos1 = chunk1.getPOSs();
+		List<String> pos2 = chunk2.getPOSs();
+		List<String> lem1 = chunk1.getLemmas();
+		List<String> lem2 = chunk2.getLemmas();
+
+		List<String> ner1 = new ArrayList<String>();
+		List<String> ner2 = new ArrayList<String>();
+
+
+		for (ParseTreeNode node: chunk1.getParseTreeNodes()) {
+			if (node.getNe()!=null && !node.getNe().equals("O"))
+				ner1.add(node.getNe());
+		}
+
+		for (ParseTreeNode node: chunk2.getParseTreeNodes()) {
+			if (node.getNe()!=null && !node.getNe().equals("O"))
+				ner2.add(node.getNe());
+		}
+
+
+		List<String> overlap = new ArrayList<String>(ner1);
+		overlap.retainAll(ner2);
+		overlap = new ArrayList<String>(new HashSet<String>(overlap));
+
+
+		if (overlap == null || overlap.size() < 1)
+			return null;
+
+		List<Integer> occur1 = new ArrayList<Integer>(), occur2 = new ArrayList<Integer>();
+		for (String word : overlap) {
+			Integer i1 = ner1.indexOf(word);
+			Integer i2 = ner2.indexOf(word);
+			occur1.add(i1);
+			occur2.add(i2);
+		}
+
+
+		// for verbs find alignment even if no same verb lemmas, just any pair of verbs. Usually should be 0,0
+		if (chunk1.getMainPOS().startsWith("VP") && chunk2.getMainPOS().startsWith("VP")) {
+			Integer i1 = null, i2 = null;
+			for(int i=0; i< pos1.size(); i++){
+				if (pos1.get(i).startsWith("VB")){
+					i1 = i;
+					break;
+				}
+			}
+
+			for(int i=0; i< pos2.size(); i++){
+				if (pos2.get(i).startsWith("VB")){
+					i2 = i;
+					break;
+				}
+			}
+			occur1.add(i1);
+			occur2.add(i2);
+		}
+		// now we search for plausible sublists of overlaps
+		// if at some position correspondence is inverse (one of two position
+		// decreases instead of increases)
+		// then we terminate current alignment accum and start a new one
+		List<List<int[]>> overlapsPlaus = new ArrayList<List<int[]>>();
+		// starts from 1, not 0
+		List<int[]> accum = new ArrayList<int[]>();
+		accum.add(new int[] { occur1.get(0), occur2.get(0) });
+		for (int i = 1; i < occur1.size(); i++) {
+
+			if (occur1.get(i) > occur1.get(i - 1)
+					&& occur2.get(i) > occur2.get(i - 1))
+				accum.add(new int[] { occur1.get(i), occur2.get(i) });
+			else {
+				overlapsPlaus.add(accum);
+				if (occur1!=null && occur2!=null && i<occur1.size() &&  i<occur2.size() ){
+					accum = new ArrayList<int[]>();
+					accum.add(new int[] { occur1.get(i), occur2.get(i) });
+				}
+			}
+		}
+		if (accum.size() > 0) {
+			overlapsPlaus.add(accum);
+		}
+
+
+		for (List<int[]> occur : overlapsPlaus) {
+			List<Integer> occr1 = new ArrayList<Integer>(), occr2 = new ArrayList<Integer>();
+			for (int[] column : occur) {
+				occr1.add(column[0]);
+				occr2.add(column[1]);
+			}
+
+			int ov1 = 0, ov2 = 0; // iterators over common words;
+			List<String> commonPOS = new ArrayList<String>(), commonLemmas = new ArrayList<String>();
+			// we start two words before first word
+			int k1 = occr1.get(ov1) - 2, k2 = occr2.get(ov2) - 2;
+			Boolean bReachedCommonWord = false;
+			while (k1 < 0 || k2 < 0) {
+				k1++;
+				k2++;
+			}
+			int k1max = pos1.size() - 1, k2max = pos2.size() - 1;
+			while (k1 <= k1max && k2 <= k2max) {
+				/*        // first check if the same POS
+        String sim = posManager.similarPOS(pos1.get(k1), pos2.get(k2));
+        String lemmaMatch = lemmaFormManager.matchLemmas(ps, lem1.get(k1),
+            lem2.get(k2), sim);
+				 */      
+				String sim = null;
+				List<String> sims = posManager.//similarPOS(pos1.get(k1), pos2.get(k2));
+						generalize(pos1.get(k1), pos2.get(k2));
+				if (!sims.isEmpty())
+					sim = sims.get(0);
+
+				String lemmaMatch = null;		
+				List<String> lemmaMatchs = lemmaFormManager.//matchLemmas(ps, 
+						generalize(lem1.get(k1),
+								lem2.get(k2));
+				if (!lemmaMatchs.isEmpty())
+					lemmaMatch = lemmaMatchs.get(0);
+
+
+
+				if ((sim != null)
+						&& (lemmaMatch == null || (lemmaMatch != null && !lemmaMatch
+						.equals("fail")))) {
+					commonPOS.add(pos1.get(k1));
+
+
+					// doing parse tree node generalization
+					List<ParseTreeNode> genRes =  nodeGen.generalize(chunk1.getParseTreeNodes().get(k1), chunk2.getParseTreeNodes().get(k2)); 
+					if (genRes.size()==1)
+						results.add(genRes.get(0));
+
+					if (lemmaMatch != null) {
+						commonLemmas.add(lemmaMatch);
+						// System.out.println("Added "+lemmaMatch);
+						if (k1 == occr1.get(ov1) && k2 == occr2.get(ov2))
+							bReachedCommonWord = true; // now we can have different increment
+						// opera
+						else {
+							if (occr1.size() > ov1 + 1 && occr2.size() > ov2 + 1
+									&& k1 == occr1.get(ov1 + 1) && k2 == occr2.get(ov2 + 1)) {
+								ov1++;
+								ov2++;
+								bReachedCommonWord = true;
+							}
+							// else
+								// System.err.println("Next match reached '"+lemmaMatch+
+							// "' | k1 - k2: "+k1 + " "+k2 +
+							// "| occur index ov1-ov2 "+
+							// ov1+" "+ov2+
+							// "| identified positions of match: occr1.get(ov1) - occr2.get(ov1) "
+							// +
+							// occr1.get(ov1) + " "+ occr2.get(ov1));
+						}
+					} else {
+						commonLemmas.add("*");
+					} // the same parts of speech, proceed to the next word in both
+					// expressions
+					k1++;
+					k2++;
+
+				} else if (!bReachedCommonWord) {
+					k1++;
+					k2++;
+				} // still searching
+				else {
+					// different parts of speech, jump to the next identified common word
+					ov1++;
+					ov2++;
+					if (ov1 > occr1.size() - 1 || ov2 > occr2.size() - 1)
+						break;
+					// now trying to find
+					int kk1 = occr1.get(ov1) - 2, // new positions of iterators
+							kk2 = occr2.get(ov2) - 2;
+					int countMove = 0;
+					while ((kk1 < k1 + 1 || kk2 < k2 + 1) && countMove < 2) { // if it is
+						// behind
+						// current
+						// position,
+						// synchroneously
+						// move
+						// towards
+						// right
+						kk1++;
+						kk2++;
+						countMove++;
+					}
+					k1 = kk1;
+					k2 = kk2;
+
+					if (k1 > k1max)
+						k1 = k1max;
+					if (k2 > k2max)
+						k2 = k2max;
+					bReachedCommonWord = false;
+				}
+			}
+			ParseTreeChunk currResult = new ParseTreeChunk(results),
+					currResultOld = new ParseTreeChunk(commonLemmas, commonPOS, 0, 0);
+
+
+			resultChunks.add(currResult);
+		}
+
+		return resultChunks;
+	}
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/ParseTreeNodeGeneralizer.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/ParseTreeNodeGeneralizer.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/ParseTreeNodeGeneralizer.java
new file mode 100644
index 0000000..8001a7b
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/ParseTreeNodeGeneralizer.java
@@ -0,0 +1,68 @@
+package opennlp.tools.parse_thicket.matching;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+
+import opennlp.tools.parse_thicket.IGeneralizer;
+import opennlp.tools.parse_thicket.ParseTreeNode;
+import opennlp.tools.parse_thicket.VerbNetProcessor;
+
+public class ParseTreeNodeGeneralizer implements IGeneralizer<ParseTreeNode>{
+	private LemmaGeneralizer lGen = new LemmaGeneralizer ();
+	private PartOfSpeechGeneralizer posGen = new PartOfSpeechGeneralizer ();
+	private VerbNetProcessor vnProc = VerbNetProcessor.getInstance(null);
+
+	@Override
+	public List<ParseTreeNode> generalize(Object o1, Object o2) {
+
+		List<ParseTreeNode> results = new ArrayList<ParseTreeNode>();
+
+		ParseTreeNode newNode = null;
+		ParseTreeNode ch1 = (ParseTreeNode)o1, ch2 = (ParseTreeNode)o2;
+		List<String> posGenStrList =  posGen.generalize(ch1.getPos(), ch2.getPos());
+		if (!posGenStrList.isEmpty()){
+			List<String> lemmaGen = lGen.generalize(ch1.getWord(), ch2.getWord());
+			if (!lemmaGen.isEmpty())
+				newNode = new ParseTreeNode(lemmaGen.get(0),  posGenStrList.get(0), "O", -1);
+			else
+				newNode = new ParseTreeNode("*",  posGenStrList.get(0), "O", -1);
+		}
+		
+		newNode.setPhraseType(ch1.getPhraseType());
+		//TODO separate NER generalizer
+		//TODO multiword generalizer
+		if (posGenStrList.get(0).startsWith("NN")){
+			if (ch1.getNe()!=null && ch2.getNe()!=null && ch1.getNe().equals(ch2.getNe()))
+					newNode.setNe(ch1.getNe());
+		}
+		if (posGenStrList.get(0).startsWith("VB")){	
+			List<Map<String, List<String>>> verbNetGenList = vnProc .generalize(ch1.getWord(), ch2.getWord());
+			if (verbNetGenList.size()>0){
+				Map<String, List<String>> verbNetGen = verbNetGenList.get(0);
+				Map<String, Object> attr = newNode.getAttributes();
+				if (attr == null)
+					attr = new HashMap<String, Object> ();
+				try {
+					List<String> phrDscr = (List<String>) attr.get("phrDescr");
+					if (phrDscr!=null) // && phrDscr.size()>1)
+						phrDscr = new ArrayList<String>(new HashSet<String>(phrDscr));
+				} catch (Exception e) {
+					System.err.println("Problem de-duplicating verbnet expr" + attr);
+				}
+				if (verbNetGen!=null){
+					attr.putAll(verbNetGen);
+					newNode.setAttributes(attr);
+				}
+			}
+		} else if (posGenStrList.get(0).startsWith("NN")){
+			//TODO
+		}
+
+		results.add(newNode);
+		return results;
+	}
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/PartOfSpeechGeneralizer.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/PartOfSpeechGeneralizer.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/PartOfSpeechGeneralizer.java
new file mode 100644
index 0000000..66efe23
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/PartOfSpeechGeneralizer.java
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.parse_thicket.matching;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import opennlp.tools.parse_thicket.IGeneralizer;
+
+public class PartOfSpeechGeneralizer implements IGeneralizer<String>{
+
+	@Override
+	public List<String> generalize(Object o1, Object o2){
+		String pos1 = (String)o1, pos2 =  (String) o2;
+		List<String>  results = new ArrayList<String>();
+		String res = computeSimilarity(pos1, pos2);
+		if (res!=null)
+			results.add(res);
+		return results;
+
+	}
+	private String computeSimilarity(String pos1, String pos2){
+
+		if ((pos1.startsWith("NN") && pos2.equals("NP") || pos2.startsWith("NN")
+				&& pos1.equals("NP"))) {
+			return "NN";
+		}
+		if ((pos1.startsWith("NN") && pos2.equals("VBG") || pos2.startsWith("VBG")
+				&& pos1.equals("NN"))) {
+			return "NN";
+		}
+
+		if ((pos1.startsWith("NN") && pos2.equals("ADJP") || pos2.startsWith("NN")
+				&& pos1.equals("ADJP"))) {
+			return "NN";
+		}
+		if ((pos1.equals("IN") && pos2.equals("TO") || pos1.equals("TO")
+				&& pos2.equals("IN"))) {
+			return "IN";
+		}
+		// VBx vs VBx = VB (does not matter which form for verb)
+		if (pos1.startsWith("VB") && pos2.startsWith("VB")) {
+			return "VB";
+		}
+
+		// ABx vs ABy always gives AB
+		if (pos1.equalsIgnoreCase(pos2)) {
+			return pos1;
+		}
+		if (pos1.length() > 2) {
+			pos1 = pos1.substring(0, 2);
+		}
+
+		if (pos2.length() > 2) {
+			pos2 = pos2.substring(0, 2);
+		}
+		if (pos1.equalsIgnoreCase(pos2)) {
+			return pos1 + "*";
+		}
+		return null;
+	}
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/PersonalInformationExtractor.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/PersonalInformationExtractor.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/PersonalInformationExtractor.java
new file mode 100644
index 0000000..5df0dee
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/PersonalInformationExtractor.java
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.parse_thicket.matching;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import opennlp.tools.jsmlearning.ProfileReaderWriter;
+import opennlp.tools.parse_thicket.VerbNetProcessor;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.tika.Tika;
+import org.apache.tika.exception.TikaException;
+
+public class PersonalInformationExtractor {
+	FrameQueryBasedIExtractor extractor = new FrameQueryBasedIExtractor();
+	private ArrayList<File> queue = new ArrayList<File>();
+	private Tika tika = new Tika();
+
+	public void runExtractor(String filename){
+		String content = null;
+		try {
+			content = FileUtils.readFileToString(new File(filename));
+		} catch (IOException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		}
+
+
+		extractor.buildTemplates(new String[] { "John Doe send his California driver license 1234567 . "
+				+ "Jill Jones received her Ohio license 4567456. ", 
+				" Mary Poppins got her identification 8765. Jorge Malony sold his identification 9876. ",
+				//" President Jorge Smith of Microsoft used his id 4567. Manager John Smith of Google used his id 8765. "
+				" Johh Doe 123. Don Joe 1323. "
+
+		});
+
+		List<GeneralizationResult>  res = extractor.doIE( content);
+
+	}
+
+
+	private void addFiles(File file) {
+
+		try {
+			if (!file.exists()) {
+				System.out.println(file + " does not exist.");
+			}
+			if (file.isDirectory()) {
+				for (File f : file.listFiles()) {
+					try {
+						addFiles(f);
+					} catch (Exception e) {
+					}
+				}
+			} else {
+				queue.add(file);
+			}
+		} catch (Exception e) {
+
+		}
+	}
+
+	public void processDirectory(String filename, String template) throws IOException {
+		List<String[]> report = new ArrayList<String[]>(); 
+		report.add(new String[]{"filename", "text",  "generalization", "fired?" });
+		String templateStr = null;
+		try {
+
+			templateStr =  FileUtils.readFileToString(new File(template));
+		} catch (IOException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		}
+
+		String[] samples = templateStr.split("&");
+
+		extractor.buildTemplates(samples);
+
+		addFiles(new File(filename));
+
+
+		for (File f : queue) {
+			String content=null;
+			try {
+				content = tika.parseToString(f);
+				List<GeneralizationResult>  res = extractor.doIE( content);
+
+				for(GeneralizationResult gr: res){
+					report.add(new String[]{filename, gr.getText(),  gr.getGen().toString(), gr.getbFire().toString() });
+				}
+
+			} catch (TikaException e) {
+				System.out.println("Tika problem with file" + f.getAbsolutePath());
+			} catch (Exception ee){
+				ee.printStackTrace();
+			}
+			ProfileReaderWriter.writeReport(report, "PII_report.csv");
+		}
+
+		queue.clear();
+	}
+
+
+	public void runExtractor(String filename, String template){
+		String content = null, templateStr = null;
+		try {
+			content = FileUtils.readFileToString(new File(filename));
+			templateStr =  FileUtils.readFileToString(new File(template));
+		} catch (IOException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		}
+
+		String[] samples = templateStr.split("&");
+
+		extractor.buildTemplates(samples);
+
+		List<GeneralizationResult>  res = extractor.doIE( content);
+		List<String[]> report = new ArrayList<String[]>();
+
+		for(GeneralizationResult gr: res){
+			report.add(new String[]{filename, gr.getText(),  gr.getGen().toString(), gr.getbFire().toString() });
+		}
+
+
+	}
+
+	public static void main(String[] args){
+		//String filename = "/Users/borisgalitsky/Documents/workspace/deepContentInspection/src/test/resources/pii/agreement.txt";
+		
+		if (args ==null || args.length!=3)
+			System.err.println("Usage: java -Xmx10g -jar *.jar path-to-resources path-to-file-to-analyze path-to-file-with_samples\n");
+		try {
+			VerbNetProcessor.getInstance(args[0]);
+			new PersonalInformationExtractor().processDirectory( args[1], args[2]);
+		} catch (IOException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		}
+	}
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/PhraseGeneralizer.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/PhraseGeneralizer.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/PhraseGeneralizer.java
new file mode 100644
index 0000000..61d5f0f
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/PhraseGeneralizer.java
@@ -0,0 +1,268 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.parse_thicket.matching;
+
+import java.util.ArrayList;
+import java.util.List;
+
+
+
+
+
+
+import opennlp.tools.parse_thicket.IGeneralizer;
+import opennlp.tools.parse_thicket.ParseTreeNode;
+import opennlp.tools.stemmer.PStemmer;
+import opennlp.tools.textsimilarity.GeneralizationListReducer;
+import opennlp.tools.textsimilarity.LemmaFormManager;
+import opennlp.tools.textsimilarity.POSManager;
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+
+public class PhraseGeneralizer implements IGeneralizer<ParseTreeChunk> {
+
+	private GeneralizationListReducer generalizationListReducer = new GeneralizationListReducer();
+	protected LemmaGeneralizer lemmaFormManager = new LemmaGeneralizer();
+	//protected LemmaFormManager lemmaFormManager = new LemmaFormManager();
+	
+	
+	protected PartOfSpeechGeneralizer posManager = new PartOfSpeechGeneralizer();
+	
+	protected PStemmer ps = new PStemmer();
+	protected ParseTreeNodeGeneralizer nodeGen = new ParseTreeNodeGeneralizer();
+
+	/**
+	 * key matching function which takes two phrases, aligns them and finds a set
+	 * of maximum common sub-phrase
+	 * 
+	 * @param chunk1
+	 * @param chunk2
+	 * @return
+	 */
+	@Override
+	public List<ParseTreeChunk> generalize(
+			Object chunk1o, Object chunk2o) {
+
+		ParseTreeChunk chunk1 = (ParseTreeChunk)chunk1o, chunk2 = (ParseTreeChunk)chunk2o;
+
+		List<ParseTreeChunk> resultChunks = new ArrayList<ParseTreeChunk>();
+
+
+		List<String> pos1 = chunk1.getPOSs();
+		List<String> pos2 = chunk2.getPOSs();
+		List<String> lem1 = chunk1.getLemmas();
+		List<String> lem2 = chunk2.getLemmas();
+
+		List<String> lem1stem = new ArrayList<String>();
+		List<String> lem2stem = new ArrayList<String>();
+
+
+		for (String word : lem1) {
+			try {
+				lem1stem.add(ps.stem(word.toLowerCase()).toString());
+			} catch (Exception e) {
+				// e.printStackTrace();
+
+				if (word.length() > 2)
+					System.err.println("Unable to stem: " + word);
+			}
+		}
+		try {
+			for (String word : lem2) {
+				lem2stem.add(ps.stem(word.toLowerCase()).toString());
+			}
+		} catch (Exception e) {
+			System.err.println("problem processing word " + lem2.toString());
+		}
+
+		List<String> overlap = new ArrayList<String>(lem1stem);
+		overlap.retainAll(lem2stem);
+
+		if (overlap == null || overlap.size() < 1)
+			return null;
+
+		// to accumulate starts of alignments
+		List<Integer> occur1 = new ArrayList<Integer>(), occur2 = new ArrayList<Integer>();
+
+		// for verbs find alignment even if no same verb lemmas, just any pair of verbs. Usually should be 0,0
+		if (chunk1.getMainPOS().startsWith("VP") && chunk2.getMainPOS().startsWith("VP")) {
+			Integer i1 = null, i2 = null;
+			for(int i=0; i< pos1.size(); i++){
+				if (pos1.get(i).startsWith("VB")){
+					i1 = i;
+					break;
+				}
+			}
+
+			for(int i=0; i< pos2.size(); i++){
+				if (pos2.get(i).startsWith("VB")){
+					i2 = i;
+					break;
+				}
+			}
+			if (i1!=null)
+				occur1.add(i1);
+			if (i2!=null)
+				occur2.add(i2);
+		}
+
+
+		for (String word : overlap) {
+			Integer i1 = lem1stem.indexOf(word);
+			Integer i2 = lem2stem.indexOf(word);
+			occur1.add(i1);
+			occur2.add(i2);
+		}
+
+
+		// now we search for plausible sublists of overlaps
+		// if at some position correspondence is inverse (one of two position
+		// decreases instead of increases)
+		// then we terminate current alignment accum and start a new one
+		List<List<int[]>> overlapsPlaus = new ArrayList<List<int[]>>();
+		// starts from 1, not 0
+		List<int[]> accum = new ArrayList<int[]>();
+		accum.add(new int[] { occur1.get(0), occur2.get(0) });
+		for (int i = 1; i < occur1.size() && i< occur2.size(); i++) {
+
+			if (occur1.get(i) > occur1.get(i - 1)
+					&& occur2.get(i) > occur2.get(i - 1))
+				accum.add(new int[] { occur1.get(i), occur2.get(i) });
+			else {
+				overlapsPlaus.add(accum);
+				accum = new ArrayList<int[]>();
+				accum.add(new int[] { occur1.get(i), occur2.get(i) });
+			}
+		}
+		if (accum.size() > 0) {
+			overlapsPlaus.add(accum);
+		}
+
+
+		for (List<int[]> occur : overlapsPlaus) {
+			List<ParseTreeNode> results = new ArrayList<ParseTreeNode>();
+			List<Integer> occr1 = new ArrayList<Integer>(), occr2 = new ArrayList<Integer>();
+			for (int[] column : occur) {
+				occr1.add(column[0]);
+				occr2.add(column[1]);
+			}
+
+			int ov1 = 0, ov2 = 0; // iterators over common words;
+			List<String> commonPOS = new ArrayList<String>(), commonLemmas = new ArrayList<String>();
+			// we start two words before first word
+			int k1 = occr1.get(ov1) - 2, k2 = occr2.get(ov2) - 2;
+			// if (k1<0) k1=0; if (k2<0) k2=0;
+			Boolean bReachedCommonWord = false;
+			while (k1 < 0 || k2 < 0) {
+				k1++;
+				k2++;
+			}
+			int k1max = pos1.size() - 1, k2max = pos2.size() - 1;
+			while (k1 <= k1max && k2 <= k2max) {
+				// first check if the same POS
+				String sim = null;
+				List<String> sims = posManager.//similarPOS(pos1.get(k1), pos2.get(k2));
+						generalize(pos1.get(k1), pos2.get(k2));
+				if (!sims.isEmpty())
+					sim = sims.get(0);
+				
+				String lemmaMatch = null;		
+				List<String> lemmaMatchs = lemmaFormManager.//matchLemmas(ps, 
+						generalize(lem1.get(k1),
+						lem2.get(k2));
+				if (!lemmaMatchs.isEmpty())
+					lemmaMatch = lemmaMatchs.get(0);
+				
+
+				if ((sim != null)
+						&& (lemmaMatch == null || (lemmaMatch != null ))) {
+					commonPOS.add(pos1.get(k1));
+
+
+					// doing parse tree node generalization
+					List<ParseTreeNode> genRes =  nodeGen.generalize(chunk1.getParseTreeNodes().get(k1), chunk2.getParseTreeNodes().get(k2)); 
+					if (genRes.size()==1)
+						results.add(genRes.get(0));
+
+					if (lemmaMatch != null) {
+						commonLemmas.add(lemmaMatch);
+						// System.out.println("Added "+lemmaMatch);
+						if (k1 == occr1.get(ov1) && k2 == occr2.get(ov2))
+							bReachedCommonWord = true; // now we can have different increment
+						// opera
+						else {
+							if (occr1.size() > ov1 + 1 && occr2.size() > ov2 + 1
+									&& k1 == occr1.get(ov1 + 1) && k2 == occr2.get(ov2 + 1)) {
+								ov1++;
+								ov2++;
+								bReachedCommonWord = true;
+							}
+						}
+					} else {
+						commonLemmas.add("*");
+					} // the same parts of speech, proceed to the next word in both
+					// expressions
+					k1++;
+					k2++;
+
+				} else if (!bReachedCommonWord) {
+					k1++;
+					k2++;
+				} // still searching
+				else {
+					// different parts of speech, jump to the next identified common word
+					ov1++;
+					ov2++;
+					if (ov1 > occr1.size() - 1 || ov2 > occr2.size() - 1)
+						break;
+					// now trying to find
+					int kk1 = occr1.get(ov1) - 2, // new positions of iterators
+							kk2 = occr2.get(ov2) - 2;
+					int countMove = 0;
+					while ((kk1 < k1 + 1 || kk2 < k2 + 1) && countMove < 2) { // if it is
+						// behind
+						// current
+						// position,
+						// synchroneously
+						// move
+						// towards
+						// right
+						kk1++;
+						kk2++;
+						countMove++;
+					}
+					k1 = kk1;
+					k2 = kk2;
+
+					if (k1 > k1max)
+						k1 = k1max;
+					if (k2 > k2max)
+						k2 = k2max;
+					bReachedCommonWord = false;
+				}
+			}
+			ParseTreeChunk currResult = new ParseTreeChunk(results);
+			//currResultOld = new ParseTreeChunk(commonLemmas, commonPOS, 0, 0);
+
+
+			resultChunks.add(currResult);
+		}
+
+		return resultChunks;
+	}
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/PhraseGroupGeneralizer.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/PhraseGroupGeneralizer.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/PhraseGroupGeneralizer.java
new file mode 100644
index 0000000..094f093
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/matching/PhraseGroupGeneralizer.java
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.parse_thicket.matching;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import opennlp.tools.parse_thicket.IGeneralizer;
+import opennlp.tools.parse_thicket.ParseTreeNode;
+import opennlp.tools.stemmer.PStemmer;
+import opennlp.tools.textsimilarity.GeneralizationListReducer;
+import opennlp.tools.textsimilarity.LemmaFormManager;
+import opennlp.tools.textsimilarity.POSManager;
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+
+public class PhraseGroupGeneralizer implements IGeneralizer<List<ParseTreeChunk>>{
+
+  private GeneralizationListReducer generalizationListReducer = new GeneralizationListReducer();
+
+  private LemmaFormManager lemmaFormManager = new LemmaFormManager();
+
+  private POSManager posManager = new POSManager();
+
+  private PhraseGeneralizer pGen = new PhraseGeneralizer();
+  private NERPhraseGeneralizer pGenNER = new NERPhraseGeneralizer();
+
+  /**
+   * main function to generalize two expressions grouped by phrase types returns
+   * a list of generalizations for each phrase type with filtered
+   * sub-expressions
+   * 
+   * @param sent1
+   * @param sent2
+   * @return List<List<ParseTreeChunk>> list of list of POS-words pairs for each
+   *         resultant matched / overlapped phrase
+   */
+  @Override
+  public List<List<ParseTreeChunk>> generalize(Object o1, Object o2) {
+	  
+  
+      List<List<ParseTreeChunk>> sent1 = (List<List<ParseTreeChunk>>)o1, 
+    	 sent2 = (List<List<ParseTreeChunk>>) o2 ;
+    List<List<ParseTreeChunk>> results = new ArrayList<List<ParseTreeChunk>>();
+    // first iterate through component
+    for (int comp = 0; comp < 2 && // just np & vp
+        comp < sent1.size() && comp < sent2.size(); comp++) {
+      List<ParseTreeChunk> resultComps = new ArrayList<ParseTreeChunk>();
+      // then iterate through each phrase in each component
+      // first try lemma-based alignment
+      for (ParseTreeChunk ch1 : sent1.get(comp)) {
+        for (ParseTreeChunk ch2 : sent2.get(comp)) { // simpler version
+          List<ParseTreeChunk> chunkToAdd=null;
+		try {
+			chunkToAdd = pGen.generalize(ch1, ch2);
+		} catch (Exception e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		}
+          if (chunkToAdd == null){
+        	     chunkToAdd = new ArrayList<ParseTreeChunk>();
+          }       
+          Boolean alreadyThere = false;
+          for (ParseTreeChunk chunk : resultComps) {
+            if (chunkToAdd.contains(chunk)) {
+              alreadyThere = true;
+              break;
+            }
+          }
+
+          if (!alreadyThere && chunkToAdd != null && chunkToAdd.size() > 0) {
+            resultComps.addAll(chunkToAdd);
+          }
+
+        }
+      } // then try NER-based alignment
+      if (comp==0 || resultComps.size()<1){
+    	  for (ParseTreeChunk ch1 : sent1.get(comp)) {
+    	        for (ParseTreeChunk ch2 : sent2.get(comp)) { // simpler version
+    	          List<ParseTreeChunk> chunkToAdd = pGenNER.generalize(
+    	              ch1, ch2);
+
+    	          if (chunkToAdd == null){
+    	        	   chunkToAdd = new ArrayList<ParseTreeChunk>();
+    	          }
+    	         
+    	          Boolean alreadyThere = false;
+    	          for (ParseTreeChunk chunk : resultComps) {
+    	            if (chunkToAdd.contains(chunk)) {
+    	              alreadyThere = true;
+    	              break;
+    	            }
+    	          }
+
+    	          if (!alreadyThere && chunkToAdd != null && chunkToAdd.size() > 0) {
+    	            resultComps.addAll(chunkToAdd);
+    	          }
+
+    	        }
+    	      }
+      }
+      
+      List<ParseTreeChunk> resultCompsRed = generalizationListReducer.applyFilteringBySubsumption(resultComps);
+
+      resultComps = resultCompsRed;
+      results.add(resultComps);
+    }
+
+    return results;
+  }
+
+  
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/AbstractEngineRunner.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/AbstractEngineRunner.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/AbstractEngineRunner.java
new file mode 100644
index 0000000..ba8a140
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/AbstractEngineRunner.java
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.parse_thicket.opinion_processor;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.List;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+
+import opennlp.tools.jsmlearning.ProfileReaderWriter;
+import opennlp.tools.parse_thicket.ParseTreeNode;
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+
+public class AbstractEngineRunner {
+	private List<File> queue;
+	private final static String reviewSource = "/Users/bgalitsky/Documents/relevance-based-on-parse-trees/src/test/resources/opinions/macbook_pro.txt";
+	NamedEntityExtractor neExtractor = new NamedEntityExtractor();
+	
+	public void processJSONfileWithReviews(){
+		List<String[]> report = new ArrayList<String[]>();
+		report.add(new String[] { "text", "phrases of potential interest list" , });
+
+		
+		String content=null;
+		try {
+			content = FileUtils.readFileToString(new File(reviewSource));
+		} catch (IOException e) {
+			e.printStackTrace();
+		}
+		String[] texts = StringUtils.substringsBetween(content, "reviewText\": \"", "\", \"overall");
+		for(String text: texts){
+			EntityExtractionResult result = neExtractor.extractEntities(text);
+			report.add(new String[]{text});
+			//report.add((String[])result.extractedNERWords.toArray(new String[0]));
+			//report.add((String[])result.extractedSentimentPhrases.toArray(new String[0]));
+			List<String> stringPhrases = new ArrayList<String>(),
+					nodePhrases = new ArrayList<String>();
+			for(List<ParseTreeNode> chList: result.extractedSentimentPhrases){
+				String buf = "", nodeBuf="";
+				for(ParseTreeNode ch: chList){
+					buf+=ch.getWord()+ " ";
+					nodeBuf+=ch.toString()+ " ";
+				}
+				stringPhrases.add(buf.trim());
+				nodePhrases.add(nodeBuf.trim());
+			}
+			report.add((String[])stringPhrases.toArray(new String[0]));
+			report.add((String[])nodePhrases.toArray(new String[0]));
+			report.add(new String[]{"-----------------------------"});
+			ProfileReaderWriter.writeReport(report, "nameEntitiesTopicsOfInterestExtracted.csv");
+		}
+	}
+
+	// this func collects files 
+		private void addFiles(File file) {
+
+			if (!file.exists()) {
+				System.out.println(file + " does not exist.");
+			}
+			if (file.isDirectory()) {
+				for (File f : file.listFiles()) {
+					if (f.getName().startsWith("."))
+						continue;
+					addFiles(f);
+					System.out.println(f.getName());
+				}
+			} else {
+				queue.add(file);
+
+			}
+		}
+	
+	public static void main(String[] args){
+		AbstractEngineRunner runner = new AbstractEngineRunner();
+		runner.processJSONfileWithReviews();
+
+	}
+}
+
+/*
+	public void processDirectory(String path){
+		List<String[]> report = new ArrayList<String[]>();
+		report.add(new String[] { "filename", "named entity list", "phrases of potential interest list" });
+
+		List<String> allNamedEntities = new ArrayList<String>();
+
+		addFiles(new File(path));
+		for(File f: queue){
+			List<String> entities = (List<String>) extractEntities(f.getAbsolutePath()).getFirst();
+			List<String> opinions = (List<String>) extractEntities(f.getAbsolutePath()).getSecond();
+			report.add(new String[]{ f.getName(), entities.toString(),  opinions.toString()});	
+			ProfileReaderWriter.writeReport(report, "nameEntitiesExtracted.csv");
+
+			allNamedEntities.addAll(entities);
+
+			allNamedEntities = new ArrayList<String>(new HashSet<String> (allNamedEntities ));
+
+
+		}
+		ProfileReaderWriter.writeReport(report, "nameEntitiesTopicsOfInterestExtracted.csv");
+	} 
+} */

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/DefaultSentimentProcessor.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/DefaultSentimentProcessor.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/DefaultSentimentProcessor.java
new file mode 100644
index 0000000..44a3640
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/DefaultSentimentProcessor.java
@@ -0,0 +1,523 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.parse_thicket.opinion_processor;
+
+import java.io.IOException;
+import java.util.List;
+
+import edu.stanford.nlp.util.logging.Redwood;
+
+import java.util.Iterator;
+import java.io.BufferedReader;
+import java.io.IOException;
+import java.io.FileOutputStream;
+import java.io.PrintStream;
+import java.text.DecimalFormat;
+import java.text.NumberFormat;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+import java.util.logging.Logger;
+
+import org.ejml.simple.SimpleMatrix;
+
+import edu.stanford.nlp.io.IOUtils;
+import edu.stanford.nlp.ling.CoreAnnotations;
+import edu.stanford.nlp.ling.CoreLabel;
+import edu.stanford.nlp.ling.Label;
+import edu.stanford.nlp.ling.LabeledWord;
+import edu.stanford.nlp.ling.TaggedWord;
+import edu.stanford.nlp.ling.WordLemmaTag;
+import edu.stanford.nlp.ling.WordTag;
+import edu.stanford.nlp.neural.rnn.RNNCoreAnnotations;
+import edu.stanford.nlp.pipeline.Annotation;
+import edu.stanford.nlp.pipeline.StanfordCoreNLP;
+import edu.stanford.nlp.sentiment.SentimentCoreAnnotations.SentimentAnnotatedTree;
+import edu.stanford.nlp.sentiment.SentimentUtils;
+import edu.stanford.nlp.trees.MemoryTreebank;
+import edu.stanford.nlp.trees.Tree;
+import edu.stanford.nlp.trees.TreeCoreAnnotations;
+import edu.stanford.nlp.util.CoreMap;
+import edu.stanford.nlp.util.Generics;
+import edu.stanford.nlp.ling.CoreAnnotations;
+import edu.stanford.nlp.pipeline.Annotation;
+import edu.stanford.nlp.pipeline.StanfordCoreNLP;
+import edu.stanford.nlp.sentiment.SentimentCoreAnnotations;
+import edu.stanford.nlp.trees.Tree;
+import edu.stanford.nlp.util.CoreMap;
+
+public class DefaultSentimentProcessor {
+	/** A logger for this class */
+	private static final Logger log = Logger
+			.getLogger("opennlp.tools.parse_thicket.opinion_processor.DefaultSentimentProcessor");
+
+	private static final NumberFormat NF = new DecimalFormat("0.0000");
+
+	enum Output {
+		PENNTREES, VECTORS, ROOT, PROBABILITIES
+	}
+
+	enum Input {
+		TEXT, TREES
+	}
+
+	/**
+	 * Sets the labels on the tree (except the leaves) to be the integer
+	 * value of the sentiment prediction.  Makes it easy to print out
+	 * with Tree.toString()
+	 */
+	static void setSentimentLabels(Tree tree) {
+		if (tree.isLeaf()) {
+			return;
+		}
+
+		for (Tree child : tree.children()) {
+			setSentimentLabels(child);
+		}
+
+		Label label = tree.label();
+		if (!(label instanceof CoreLabel)) {
+			throw new IllegalArgumentException("Required a tree with CoreLabels");
+		}
+		CoreLabel cl = (CoreLabel) label;
+		cl.setValue(Integer.toString(RNNCoreAnnotations.getPredictedClass(tree)));
+	}
+
+	/**
+	 * Sets the labels on the tree to be the indices of the nodes.
+	 * Starts counting at the root and does a postorder traversal.
+	 */
+	static int setIndexLabels(Tree tree, int index) {
+		if (tree.isLeaf()) {
+			return index;
+		}
+
+		tree.label().setValue(Integer.toString(index));
+		index++;
+		for (Tree child : tree.children()) {
+			index = setIndexLabels(child, index);
+		}
+		return index;
+	}
+
+	/**
+	 * Outputs the vectors from the tree.  Counts the tree nodes the
+	 * same as setIndexLabels.
+	 */
+	static int outputTreeVectors(PrintStream out, Tree tree, int index) {
+		if (tree.isLeaf()) {
+			return index;
+		}
+
+		out.print("  " + index + ":");
+		SimpleMatrix vector = RNNCoreAnnotations.getNodeVector(tree);
+		for (int i = 0; i < vector.getNumElements(); ++i) {
+			out.print("  " + NF.format(vector.get(i)));
+		}
+		out.println();
+		index++;
+		for (Tree child : tree.children()) {
+			index = outputTreeVectors(out, child, index);
+		}
+		return index;
+	}
+
+	/**
+	 * Outputs the scores from the tree.  Counts the tree nodes the
+	 * same as setIndexLabels.
+	 */
+	static int outputTreeScores(PrintStream out, Tree tree, int index) {
+		if (tree.isLeaf()) {
+			return index;
+		}
+
+		out.print("  " + index + ":");
+		SimpleMatrix vector = RNNCoreAnnotations.getPredictions(tree);
+		for (int i = 0; i < vector.getNumElements(); ++i) {
+			out.print("  " + NF.format(vector.get(i)));
+		}
+		out.println();
+		index++;
+		for (Tree child : tree.children()) {
+			index = outputTreeScores(out, child, index);
+		}
+		return index;
+	}
+
+	public static <T> String wordToString(T o, final boolean justValue) {
+		return wordToString(o, justValue, null);
+	}
+
+	public static <T> String wordToString(T o, final boolean justValue,
+			final String separator) {
+		if (justValue && o instanceof Label) {
+			if (o instanceof CoreLabel) {
+				CoreLabel l = (CoreLabel) o;
+				String w = l.value();
+				if (w == null)
+					w = l.word();
+				return w;
+			} else {
+				return (((Label) o).value());
+			}
+		} else if (o instanceof CoreLabel) {
+			CoreLabel l = ((CoreLabel) o);
+			String w = l.value();
+			if (w == null)
+				w = l.word();
+			if (l.tag() != null) {
+				if (separator == null) {
+					return w + CoreLabel.TAG_SEPARATOR + l.tag();
+				} else {
+					return w + separator + l.tag();
+				}
+			}
+			return w;
+			// an interface that covered these next four cases would be
+			// nice, but we're moving away from these data types anyway
+		} else if (separator != null && o instanceof TaggedWord) {
+			return ((TaggedWord) o).toString(separator);
+		} else if (separator != null && o instanceof LabeledWord) {
+			return ((LabeledWord) o).toString();
+		} else if (separator != null && o instanceof WordLemmaTag) {
+			return ((WordLemmaTag) o).toString(separator);
+		} else if (separator != null && o instanceof WordTag) {
+			return ((WordTag) o).toString(separator);
+		} else {
+			return (o.toString());
+		}
+	}
+
+
+	/**
+	 * Returns the sentence as a string with a space between words.
+	 * It prints out the {@code value()} of each item -
+	 * this will give the expected answer for a short form representation
+	 * of the "sentence" over a range of cases.  It is equivalent to
+	 * calling {@code toString(true)}.
+	 *
+	 * TODO: Sentence used to be a subclass of ArrayList, with this
+	 * method as the toString.  Therefore, there may be instances of
+	 * ArrayList being printed that expect this method to be used.
+	 *
+	 * @param list The tokenized sentence to print out
+	 * @return The tokenized sentence as a String
+	 */
+	public static <T> String listToString(List<T> list) {
+		return listToString(list, true);
+	}
+	/**
+	 * Returns the sentence as a string with a space between words.
+	 * Designed to work robustly, even if the elements stored in the
+	 * 'Sentence' are not of type Label.
+	 *
+	 * This one uses the default separators for any word type that uses
+	 * separators, such as TaggedWord.
+	 *
+	 * @param list The tokenized sentence to print out
+	 * @param justValue If {@code true} and the elements are of type
+	 *                  {@code Label}, return just the
+	 *                  {@code value()} of the {@code Label} of each word;
+	 *                  otherwise,
+	 *                  call the {@code toString()} method on each item.
+	 * @return The sentence in String form
+	 */
+	public static <T> String listToString(List<T> list, final boolean justValue) {
+		return listToString(list, justValue, null);
+	}
+
+	/**
+	 * As already described, but if separator is not null, then objects
+	 * such as TaggedWord
+	 *
+	 * @param separator The string used to separate Word and Tag
+	 *                  in TaggedWord, etc
+	 */
+	public static <T> String listToString(List<T> list, final boolean justValue,
+			final String separator) {
+		StringBuilder s = new StringBuilder();
+		for (Iterator<T> wordIterator = list.iterator(); wordIterator.hasNext();) {
+			T o = wordIterator.next();
+			s.append(wordToString(o, justValue, separator));
+			if (wordIterator.hasNext()) {
+				s.append(' ');
+			}
+		}
+		return s.toString();
+	}
+
+	/**
+	 * Outputs a tree using the output style requested
+	 */
+	static void outputTree(PrintStream out, CoreMap sentence, List<Output> outputFormats) {
+		Tree tree = sentence.get(SentimentCoreAnnotations.SentimentAnnotatedTree.class);
+		for (Output output : outputFormats) {
+			switch (output) {
+			case PENNTREES: {
+				Tree copy = tree.deepCopy();
+				setSentimentLabels(copy);
+				out.println(copy);
+				break;
+			}
+			case VECTORS: {
+				Tree copy = tree.deepCopy();
+				setIndexLabels(copy, 0);
+				out.println(copy);
+				outputTreeVectors(out, tree, 0);
+				break;
+			}
+			case ROOT: {
+				out.println("  " + sentence.get(SentimentCoreAnnotations.SentimentClass.class));
+				break;
+			}
+			case PROBABILITIES: {
+				Tree copy = tree.deepCopy();
+				setIndexLabels(copy, 0);
+				out.println(copy);
+				outputTreeScores(out, tree, 0);
+				break;
+			}
+			default:
+				throw new IllegalArgumentException("Unknown output format " + output);
+			}
+		}
+	}
+
+	/**
+	 * Reads an annotation from the given filename using the requested input.
+	 */
+	public static List<Annotation> getAnnotations(StanfordCoreNLP tokenizer, Input inputFormat, String filename, boolean filterUnknown) {
+		switch (inputFormat) {
+		case TEXT: {
+			String text = IOUtils.slurpFileNoExceptions(filename);
+			Annotation annotation = new Annotation(text);
+			tokenizer.annotate(annotation);
+			List<Annotation> annotations = Generics.newArrayList();
+			for (CoreMap sentence : annotation.get(CoreAnnotations.SentencesAnnotation.class)) {
+				Annotation nextAnnotation = new Annotation(sentence.get(CoreAnnotations.TextAnnotation.class));
+				nextAnnotation.set(CoreAnnotations.SentencesAnnotation.class, Collections.singletonList(sentence));
+				annotations.add(nextAnnotation);
+			}
+			return annotations;
+		}
+		case TREES: {
+			List<Tree> trees;
+			if (filterUnknown) {
+				trees = SentimentUtils.readTreesWithGoldLabels(filename);
+				trees = SentimentUtils.filterUnknownRoots(trees);
+			} else {
+				trees = Generics.newArrayList();
+				MemoryTreebank treebank = new MemoryTreebank("utf-8");
+				treebank.loadPath(filename, null);
+				for (Tree tree : treebank) {
+					trees.add(tree);
+				}
+			}
+
+			List<Annotation> annotations = Generics.newArrayList();
+			for (Tree tree : trees) {
+				CoreMap sentence = new Annotation(listToString(tree.yield()));
+				sentence.set(TreeCoreAnnotations.TreeAnnotation.class, tree);
+				List<CoreMap> sentences = Collections.singletonList(sentence);
+				Annotation annotation = new Annotation("");
+				annotation.set(CoreAnnotations.SentencesAnnotation.class, sentences);
+				annotations.add(annotation);
+			}
+			return annotations;
+		}
+		default:
+			throw new IllegalArgumentException("Unknown format " + inputFormat);
+		}
+	}
+
+	/** Runs the tree-based sentiment model on some text. */
+	public void processTextWithArgs(String[] args) throws IOException {
+		String parserModel = null;
+		String sentimentModel = null;
+
+		String filename = null;
+		String fileList = null;
+		boolean stdin = false;
+
+		boolean filterUnknown = false;
+
+		List<Output> outputFormats = Collections.singletonList(Output.ROOT);
+		Input inputFormat = Input.TEXT;
+
+		String tlppClass = "DEFAULT_TLPP_CLASS";
+
+		for (int argIndex = 0; argIndex < args.length; ) {
+			if (args[argIndex].equalsIgnoreCase("-sentimentModel")) {
+				sentimentModel = args[argIndex + 1];
+				argIndex += 2;
+			} else if (args[argIndex].equalsIgnoreCase("-parserModel")) {
+				parserModel = args[argIndex + 1];
+				argIndex += 2;
+			} else if (args[argIndex].equalsIgnoreCase("-file")) {
+				filename = args[argIndex + 1];
+				argIndex += 2;
+			} else if (args[argIndex].equalsIgnoreCase("-fileList")) {
+				fileList = args[argIndex + 1];
+				argIndex += 2;
+			} else if (args[argIndex].equalsIgnoreCase("-stdin")) {
+				stdin = true;
+				argIndex++;
+			} else if (args[argIndex].equalsIgnoreCase("-input")) {
+				inputFormat = Input.valueOf(args[argIndex + 1].toUpperCase());
+				argIndex += 2;
+			} else if (args[argIndex].equalsIgnoreCase("-output")) {
+				String[] formats = args[argIndex + 1].split(",");
+				outputFormats = new ArrayList<>();
+				for (String format : formats) {
+					outputFormats.add(Output.valueOf(format.toUpperCase()));
+				}
+				argIndex += 2;
+			} else if (args[argIndex].equalsIgnoreCase("-filterUnknown")) {
+				filterUnknown = true;
+				argIndex++;
+			} else if (args[argIndex].equalsIgnoreCase("-tlppClass")) {
+				tlppClass = args[argIndex + 1];
+				argIndex += 2;
+			} else if (args[argIndex].equalsIgnoreCase("-help")) {
+				System.exit(0);
+			} else {
+				log.info("Unknown argument " + args[argIndex + 1]);
+				throw new IllegalArgumentException("Unknown argument " + args[argIndex + 1]);
+			}
+		}
+
+		// We construct two pipelines.  One handles tokenization, if
+		// necessary.  The other takes tokenized sentences and converts
+		// them to sentiment trees.
+		Properties pipelineProps = new Properties();
+		Properties tokenizerProps = null;
+		if (sentimentModel != null) {
+			pipelineProps.setProperty("sentiment.model", sentimentModel);
+		}
+		if (parserModel != null) {
+			pipelineProps.setProperty("parse.model", parserModel);
+		}
+		if (inputFormat == Input.TREES) {
+			pipelineProps.setProperty("annotators", "binarizer, sentiment");
+			pipelineProps.setProperty("customAnnotatorClass.binarizer", "edu.stanford.nlp.pipeline.BinarizerAnnotator");
+			pipelineProps.setProperty("binarizer.tlppClass", tlppClass);
+			pipelineProps.setProperty("enforceRequirements", "false");
+		} else {
+			pipelineProps.setProperty("annotators", "parse, sentiment");
+			pipelineProps.setProperty("enforceRequirements", "false");
+			tokenizerProps = new Properties();
+			tokenizerProps.setProperty("annotators", "tokenize, ssplit");
+		}
+
+		if (stdin && tokenizerProps != null) {
+			tokenizerProps.setProperty(StanfordCoreNLP.NEWLINE_SPLITTER_PROPERTY, "true");
+		}
+
+		int count = 0;
+		if (filename != null) count++;
+		if (fileList != null) count++;
+		if (stdin) count++;
+		if (count > 1) {
+			throw new IllegalArgumentException("Please only specify one of -file, -fileList or -stdin");
+		}
+		if (count == 0) {
+			throw new IllegalArgumentException("Please specify either -file, -fileList or -stdin");
+		}
+
+		StanfordCoreNLP tokenizer = (tokenizerProps == null) ? null : new StanfordCoreNLP(tokenizerProps);
+		StanfordCoreNLP pipeline = new StanfordCoreNLP(pipelineProps);
+
+		if (filename != null) {
+			// Process a file.  The pipeline will do tokenization, which
+			// means it will split it into sentences as best as possible
+			// with the tokenizer.
+			List<Annotation> annotations = getAnnotations(tokenizer, inputFormat, filename, filterUnknown);
+			for (Annotation annotation : annotations) {
+				pipeline.annotate(annotation);
+
+				for (CoreMap sentence : annotation.get(CoreAnnotations.SentencesAnnotation.class)) {
+					System.out.println(sentence);
+					outputTree(System.out, sentence, outputFormats);
+				}
+			}
+		} else if (fileList != null) {
+			// Process multiple files.  The pipeline will do tokenization,
+			// which means it will split it into sentences as best as
+			// possible with the tokenizer.  Output will go to filename.out
+			// for each file.
+			for (String file : fileList.split(",")) {
+				List<Annotation> annotations = getAnnotations(tokenizer, inputFormat, file, filterUnknown);
+				FileOutputStream fout = new FileOutputStream(file + ".out");
+				PrintStream pout = new PrintStream(fout);
+				for (Annotation annotation : annotations) {
+					pipeline.annotate(annotation);
+
+					for (CoreMap sentence : annotation.get(CoreAnnotations.SentencesAnnotation.class)) {
+						pout.println(sentence);
+						outputTree(pout, sentence, outputFormats);
+					}
+				}
+				pout.flush();
+				fout.close();
+			}
+		} else {
+			// Process stdin.  Each line will be treated as a single sentence.
+			log.info("Reading in text from stdin.");
+			log.info("Please enter one sentence per line.");
+			log.info("Processing will end when EOF is reached.");
+			BufferedReader reader = IOUtils.readerFromStdin("utf-8");
+
+			for (String line; (line = reader.readLine()) != null; ) {
+				line = line.trim();
+				if ( ! line.isEmpty()) {
+					Annotation annotation = tokenizer.process(line);
+					pipeline.annotate(annotation);
+					for (CoreMap sentence : annotation.get(CoreAnnotations.SentencesAnnotation.class)) {
+						outputTree(System.out, sentence, outputFormats);
+					}
+				} else {
+					// Output blank lines for blank lines so the tool can be
+					// used for line-by-line text processing
+					System.out.println();
+				}
+			}
+
+		}
+	}
+
+	public float getNumericSentimentValue(String expression) {
+		Properties props = new Properties();
+		props.setProperty("annotators", "tokenize, ssplit, parse, sentiment");
+		StanfordCoreNLP pipeline = new StanfordCoreNLP(props);
+		int mainSentiment = 0;
+		if (expression != null && expression.length() > 0) {
+			int longest = 0;
+			Annotation annotation = pipeline.process(expression);
+			for (CoreMap sentence : annotation.get(CoreAnnotations.SentencesAnnotation.class)) {
+				Tree tree = sentence.get(SentimentAnnotatedTree.class);
+				int sentiment = RNNCoreAnnotations.getPredictedClass(tree);
+				String partText = sentence.toString();
+				if (partText.length() > longest) {
+					mainSentiment = sentiment;
+					longest = partText.length();
+				}
+			}
+		}
+		return mainSentiment;
+	}
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/EntityExtractionResult.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/EntityExtractionResult.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/EntityExtractionResult.java
new file mode 100644
index 0000000..69eae1d
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/EntityExtractionResult.java
@@ -0,0 +1,158 @@
+package opennlp.tools.parse_thicket.opinion_processor;
+
+import java.util.List;
+
+import opennlp.tools.parse_thicket.ParseTreeNode;
+import opennlp.tools.similarity.apps.HitBase;
+
+public class EntityExtractionResult {
+	List<List<ParseTreeNode>> extractedNERs;
+	public List<String> extractedNERWords;
+	// phrases w/sentiments
+	public List<List<ParseTreeNode>> extractedSentimentPhrases;
+	public List<String> extractedSentimentPhrasesStr;
+	// phrases w/o sentiments
+	public List<List<ParseTreeNode>> extractedNONSentimentPhrases;
+	public List<String> extractedNONSentimentPhrasesStr;
+	public List<Float> sentimentProfile;
+	
+	
+	public List<String> getExtractedSentimentPhrasesStr() {
+		return extractedSentimentPhrasesStr;
+	}
+
+	public void setExtractedSentimentPhrasesStr(List<String> extractedSentimentPhrasesStr) {
+		this.extractedSentimentPhrasesStr = extractedSentimentPhrasesStr;
+	}
+	/*
+	 * Licensed to the Apache Software Foundation (ASF) under one or more
+	 * contributor license agreements.  See the NOTICE file distributed with
+	 * this work for additional information regarding copyright ownership.
+	 * The ASF licenses this file to You under the Apache License, Version 2.0
+	 * (the "License"); you may not use this file except in compliance with
+	 * the License. You may obtain a copy of the License at
+	 *
+	 *     http://www.apache.org/licenses/LICENSE-2.0
+	 *
+	 * Unless required by applicable law or agreed to in writing, software
+	 * distributed under the License is distributed on an "AS IS" BASIS,
+	 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+	 * See the License for the specific language governing permissions and
+	 * limitations under the License.
+	 */
+	public List<List<ParseTreeNode>> getExtractedNONSentimentPhrases() {
+		return extractedNONSentimentPhrases;
+	}
+
+	public void setExtractedNONSentimentPhrases(List<List<ParseTreeNode>> extractedNONSentimentPhrases) {
+		this.extractedNONSentimentPhrases = extractedNONSentimentPhrases;
+	}
+
+	public List<String> getExtractedNONSentimentPhrasesStr() {
+		return extractedNONSentimentPhrasesStr;
+	}
+
+	public void setExtractedNONSentimentPhrasesStr(List<String> extractedNONSentimentPhrasesStr) {
+		this.extractedNONSentimentPhrasesStr = extractedNONSentimentPhrasesStr;
+	}
+
+	public List<HitBase> hits;
+	private List<List<ParseTreeNode>> extractedNerPhrases;
+	private List<String> extractedNerPhrasesStr;
+	private List<String> extractedNerPhraseTags;
+	private List<List<ParseTreeNode>> extractedNerExactPhrases;
+	private List<String> extractedNerExactStr;
+
+	public void setExtractedNERWords(List<String> extractedNERWords) {
+		this.extractedNERWords = extractedNERWords;
+	}
+
+	public void setExtractedSentimentPhrases(List<List<ParseTreeNode>> extractedSentimentPhrases) {
+		this.extractedSentimentPhrases = extractedSentimentPhrases;
+	}
+
+	public void setExtractedNER(List<List<ParseTreeNode>> extractedNERs) {
+		this.extractedNERs = extractedNERs;
+	}
+
+	public void setGossipHits(List<HitBase> hitsForAnEntity) {
+		hits = hitsForAnEntity;
+	}
+
+	public List<List<ParseTreeNode>> getExtractedNERs() {
+		return extractedNERs;
+	}
+
+	public void setExtractedNERs(List<List<ParseTreeNode>> extractedNERs) {
+		this.extractedNERs = extractedNERs;
+	}
+
+	public List<HitBase> getHits() {
+		return hits;
+	}
+
+	public void setHits(List<HitBase> hits) {
+		this.hits = hits;
+	}
+
+	public List<String> getExtractedNERWords() {
+		return extractedNERWords;
+	}
+
+	public List<List<ParseTreeNode>> getExtractedSentimentPhrases() {
+		return extractedSentimentPhrases;
+	}
+
+	public void setSentimentProfile(List<Float> sentimentProfile) {
+	    this.sentimentProfile = sentimentProfile;
+    }
+
+	public List<Float> getSentimentProfile() {
+		return sentimentProfile;
+	}
+
+	public void setExtractedNerPhrases(List<List<ParseTreeNode>> extractedNerPhrases) {
+	    this.extractedNerPhrases = extractedNerPhrases;
+	    
+    }
+
+	public void setExtractedNerPhrasesStr(List<String> extractedNerPhrasesStr) {
+	    this.extractedNerPhrasesStr = extractedNerPhrasesStr;
+	    
+    }
+
+	public List<List<ParseTreeNode>> getExtractedNerPhrases() {
+		return extractedNerPhrases;
+	}
+
+	public List<String> getExtractedNerPhrasesStr() {
+		return extractedNerPhrasesStr;
+	}
+
+	public void setExtractedNerPhraseTags(List<String> extractedNerPhraseTags) {
+	    this.extractedNerPhraseTags = extractedNerPhraseTags;	    
+    }
+
+	public List<String> getExtractedNerPhraseTags() {
+	    return this.extractedNerPhraseTags;    
+    }
+
+	public void setExtractedNerExactPhrases(List<List<ParseTreeNode>> extractedNerExactPhrases) {
+	   this.extractedNerExactPhrases = extractedNerExactPhrases;
+	    
+    }
+
+	public void setExtractedNerExactStr(List<String> extractedNerExactStr) {
+	    this.extractedNerExactStr = extractedNerExactStr;
+	    
+    }
+
+	public List<List<ParseTreeNode>> getExtractedNerExactPhrases() {
+		return extractedNerExactPhrases;
+	}
+
+	public List<String> getExtractedNerExactStr() {
+		return extractedNerExactStr;
+	}
+	
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/ExpressionSentimentAnalyzer.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/ExpressionSentimentAnalyzer.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/ExpressionSentimentAnalyzer.java
new file mode 100644
index 0000000..dc89d8b
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/ExpressionSentimentAnalyzer.java
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.parse_thicket.opinion_processor;
+
+import java.util.Properties;
+
+import edu.stanford.nlp.ling.CoreAnnotations;
+import edu.stanford.nlp.neural.rnn.RNNCoreAnnotations;
+import edu.stanford.nlp.pipeline.Annotation;
+import edu.stanford.nlp.pipeline.StanfordCoreNLP;
+import edu.stanford.nlp.sentiment.SentimentCoreAnnotations;
+import edu.stanford.nlp.sentiment.SentimentCoreAnnotations.SentimentAnnotatedTree;
+import edu.stanford.nlp.trees.Tree;
+import edu.stanford.nlp.util.CoreMap;
+
+public class ExpressionSentimentAnalyzer {
+	float findSentiment(String line) {
+		Properties props = new Properties();
+		props.setProperty("annotators", "tokenize, ssplit, parse, sentiment");
+		StanfordCoreNLP pipeline = new StanfordCoreNLP(props);
+		int mainSentiment = 0;
+		if (line != null && line.length() > 0) {
+			int longest = 0;
+			Annotation annotation = pipeline.process(line);
+			for (CoreMap sentence : annotation.get(CoreAnnotations.SentencesAnnotation.class)) {
+				Tree tree = sentence.get(SentimentAnnotatedTree.class);
+				int sentiment = RNNCoreAnnotations.getPredictedClass(tree);
+				String partText = sentence.toString();
+				if (partText.length() > longest) {
+					mainSentiment = sentiment;
+					longest = partText.length();
+				}
+			}
+		}
+		return mainSentiment;
+	}
+
+	public static void main(String[] args) {
+		float sent = new ExpressionSentimentAnalyzer().findSentiment("poor president nomee Hilary Clinton visited Mexico");
+		System.out.println(sent);
+	}
+}
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/LinguisticPhraseManager.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/LinguisticPhraseManager.java b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/LinguisticPhraseManager.java
new file mode 100644
index 0000000..0f53ec5
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/parse_thicket/opinion_processor/LinguisticPhraseManager.java
@@ -0,0 +1,591 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.parse_thicket.opinion_processor;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.commons.lang3.StringUtils;
+import opennlp.tools.jsmlearning.ProfileReaderWriter;
+import opennlp.tools.similarity.apps.BingQueryRunner;
+import opennlp.tools.similarity.apps.HitBase;
+import opennlp.tools.similarity.apps.utils.ValueSortMap;
+import opennlp.tools.stemmer.PStemmer;
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+import opennlp.tools.textsimilarity.TextProcessor;
+
+public class LinguisticPhraseManager {
+	private Map<String, Integer> freq = new ConcurrentHashMap<String, Integer>();
+	
+	// the purpose to init this static object is to show the path to resources
+	private static StopList stop = StopList.getInstance(new File(".").getAbsolutePath().replace(".","")+ "src/test/resources/");
+
+	// this list will be overwritten by the external synonyms.csv
+	private static String[][] synonymPairs = new String[][]{};
+	private PStemmer stemmer = new PStemmer();
+
+	private List<ParseTreeChunk> lingPhrases = new ArrayList<ParseTreeChunk>();
+	private List<String> standardizedTopics = new ArrayList<String>();
+	// map which shows for each ling phrase the list of ling phrases with the same head noun it belongs
+	private Map<ParseTreeChunk, List<ParseTreeChunk>> entry_group = new ConcurrentHashMap<ParseTreeChunk, List<ParseTreeChunk>>();
+
+	//  map which shows for each string phrase the list of ling phrases with the same head noun it belongs
+	private Map<String, List<ParseTreeChunk>> std_group = new ConcurrentHashMap<String, List<ParseTreeChunk>>();
+
+	private BingQueryRunner runner = new BingQueryRunner();
+	private static final int MIN_NUMBER_OF_PHRASES_TO_CONSIDER = 3;//2; 5
+	private static final int MIN_LENGTH_OF_WORD_TO_CONSIDER = 3;
+	// this function takes a log of a chain of the nodes of parse trees and builds their instances
+	// the phrases should only be VP or NP, otherwise an exception should be thrown
+	
+	
+
+	private String resourceDir;
+	public LinguisticPhraseManager(){
+		try {
+			resourceDir  = new File( "." ).getCanonicalPath()+"/src/main/resources/";
+			List<String[]> vocabs = ProfileReaderWriter.readProfiles(resourceDir+"/synonyms.csv");
+			synonymPairs = new String[vocabs.size()][2];
+			int count = 0;
+			for(String[] line: vocabs){
+				try {
+					synonymPairs[count] = line;
+					count++;
+	            } catch (Exception e) {
+	                e.printStackTrace();
+	            }
+			}
+			
+		} catch (Exception e) {
+			e.printStackTrace();
+		}		
+	}
+
+	private ParseTreeChunk parseLingPhraseIntoParseTreeChunk(String phrStr){
+		ParseTreeChunk ch = new ParseTreeChunk();
+		List<String> POSs = new ArrayList<String>(), lemmas = new ArrayList<String>();
+
+		String[] parts = phrStr.replace("]","").split(", <");
+
+		ch.setMainPOS( StringUtils.substringBetween(phrStr, ">", "'"));
+		try {
+			for(String part: parts){
+				String lemma = StringUtils.substringBetween(part, "P'", "':").toLowerCase();
+				String pos = part.substring(part.indexOf(":")+1, part.length());
+
+				if (pos==null || lemma ==null){
+					continue;
+				}
+				POSs.add(pos.trim());
+				lemmas.add(lemma.trim());
+				ch.setPOSs(POSs); ch.setLemmas(lemmas);
+			}
+		} catch (Exception e) {
+			// we expect exceptions if extracted phrases are NEITHER NP nor VP
+			// empty chunk will be given which will not create a new topic
+			e.printStackTrace();
+		}
+
+		return ch;
+	}
+
+	// this is a constructor with an array of extraction files
+	// optimized for performance
+	// only topics occurring more than MIN_NUMBER_OF_PHRASES_TO_CONSIDER times will be considered
+	public LinguisticPhraseManager(String[] loadPaths){
+		List<String[]> columns = new ArrayList<String[]>();
+		for(String file: loadPaths){
+			columns.addAll(ProfileReaderWriter.readProfiles( file));
+		}
+
+		for(String[] l: columns){
+			if (l.length<3 || l[1]==null || l[2]==null)
+				continue;
+			String word = l[1].toLowerCase().trim();
+			if (word.indexOf("=>")>-1)
+				continue;
+
+			word = isAcceptableStringPhrase(word);
+			if (word==null)
+				continue;
+
+			if (!freq.containsKey(word)) {
+				freq.put(word, 1);
+
+			} else {
+				freq.put(word, freq.get(word) + 1);
+				// once we reached the count for a topic, create it
+				if (freq.get(word)==MIN_NUMBER_OF_PHRASES_TO_CONSIDER){
+					ParseTreeChunk ch = parseLingPhraseIntoParseTreeChunk(l[2]);
+					ch = isAcceptableLingPhrase(ch);
+					if (ch==null)
+						continue;
+					lingPhrases.add(ch);
+				}
+			}		  
+		}
+		// we dont need frequency data any more
+		freq.clear();
+	}
+
+	// this is a default constructor with a single topic extraction file
+	// not optimized for performance
+	public LinguisticPhraseManager(String loadPath){
+		List<String[]> columns = ProfileReaderWriter.readProfiles( loadPath);
+		for(String[] l: columns){
+			if (l.length<3 || l[1]==null || l[2]==null)
+				continue;
+			String word = l[1].toLowerCase().trim();
+			if (word.indexOf("=>")>-1)
+				continue;
+
+			word = isAcceptableStringPhrase(word);
+			if (word==null)
+				continue;
+
+			if (!freq.containsKey(word)) {
+
+				ParseTreeChunk ch = parseLingPhraseIntoParseTreeChunk(l[2]);
+				ch = isAcceptableLingPhrase(ch);
+				if (ch==null)
+					continue;
+				freq.put(word, 1);
+				lingPhrases.add(ch);
+			} else {
+				freq.put(word, freq.get(word) + 1);
+			}		  
+
+
+		}
+		freq = ValueSortMap.sortMapByValue(freq, false);
+
+
+	}
+	// removing prepositions and articles in case it has not worked at phrase forming stage
+	private String isAcceptableStringPhrase(String word) {
+		if (word.startsWith("to "))
+			return null;
+		if (word.startsWith("a "))
+			return word.substring(2, word.length());
+
+		if (word.endsWith(" !") || word.endsWith(" ."))
+			return word.substring(0, word.length()-2).trim();
+
+		return word;
+	}
+	// we only accept NP 
+	private ParseTreeChunk isAcceptableLingPhrase(ParseTreeChunk ch) {
+		if (!ch.getMainPOS().equals("NP"))
+			return null;
+
+
+		return ch;
+	}
+
+	// groups are sets of phrases with the same head noun
+	// put all phrases in a group. Have a map from each phrase to its group: the list of members
+	public void doLingGrouping(){
+		for(int i=0; i< lingPhrases.size(); i++){
+			for(int j=i+1; j< lingPhrases.size(); j++){
+				ParseTreeChunk chI = lingPhrases.get(i);
+				ParseTreeChunk chJ = lingPhrases.get(j);
+				if (chI.getLemmas().get(chI.getLemmas().size()-1).equals(chJ.getLemmas().get(chJ.getLemmas().size()-1))
+						&& chI.getPOSs().get(chI.getLemmas().size()-1).startsWith("NN") ){
+					List<ParseTreeChunk> values = null; 
+					if( chI.getLemmas().size()<chJ.getLemmas().size()){		
+
+						if (values == null)
+							values = new ArrayList<ParseTreeChunk>();
+						values.add(chI);
+						entry_group.put(chJ, values);
+					} else {
+						values = entry_group.get(chI);
+						if (values == null)
+							values = new ArrayList<ParseTreeChunk>();
+						values.add(chJ);
+						entry_group.put(chI, values);
+					}
+				}
+			}
+		}
+
+
+	}
+
+	public List<String> formStandardizedTopic(){
+		Set<ParseTreeChunk> keys = entry_group.keySet();
+		for(ParseTreeChunk k: keys){
+			List<ParseTreeChunk> lingPhrases = entry_group.get(k);		
+			for(int i=0; i< lingPhrases.size(); i++)
+				for(int j=i+1; j< lingPhrases.size(); j++){
+					ParseTreeChunk chI = lingPhrases.get(i);
+					ParseTreeChunk chJ = lingPhrases.get(j);
+					List<String> lemmas = new ArrayList<String>(chI.getLemmas());
+					lemmas.retainAll(chJ.getLemmas());
+					if (lemmas.size()<2)
+						continue;
+					String buf = ""; List<String> candTopicLst = new ArrayList<String>();
+					for(String w: lemmas){
+						if (w.length()<MIN_LENGTH_OF_WORD_TO_CONSIDER)
+							continue;
+						if (!StringUtils.isAlpha(w))
+							continue;
+						// find POS of w
+						boolean bAccept = false;
+						for(int iw=0; iw<chI.getLemmas().size(); iw++){
+							if (w.equals(chI.getLemmas().get(iw))){
+								if (chI.getPOSs().get(iw).startsWith("NN") || chI.getPOSs().get(iw).startsWith("JJ")
+										|| chI.getPOSs().get(iw).startsWith("VB"))
+									bAccept=true;
+							}
+						}
+						if (bAccept){
+							//buf+=w+" ";
+							String ws = substituteSynonym(w);
+							candTopicLst.add(ws);
+						}
+					}
+					// remove duplicates like 'new new house'
+					//candTopicLst = new ArrayList<String>(new HashSet<String>(candTopicLst));
+					for(String w: candTopicLst){
+						buf+=w+" ";
+					}
+
+					buf = buf.trim();
+					if (buf.indexOf(' ')<0)
+						continue;
+
+					if (!standardizedTopics.contains(buf)){
+						standardizedTopics.add(buf);		
+						std_group.put(buf, lingPhrases);
+					}
+				}
+		}
+		cleanUpStandardizedTopics();
+
+		return standardizedTopics;
+	}
+
+	public void cleanUpStandardizedTopics(){
+		List<String> toDelete = new ArrayList<String>();
+		for(int i=0; i< standardizedTopics.size(); i++)
+			for(int j=i+1; j< standardizedTopics.size(); j++){
+				List<String> t1 = TextProcessor.fastTokenize(standardizedTopics.get(i), false);
+				List<String> t2 = TextProcessor.fastTokenize(standardizedTopics.get(j), false);
+				for(int k=0; k< t1.size(); k++){
+					t1.set(k, stemmer.stem(t1.get(k)));
+				}
+				for(int k=0; k< t2.size(); k++){
+					t2.set(k, stemmer.stem(t2.get(k)));
+				} 
+				// check if lists are equal
+				if (t1.size()!=t2.size())
+					continue;
+				//if in two phrases once all keywords are tokenized, one phrase annihilates another, 
+				t1.removeAll(t2);
+				if (t1.isEmpty()){ 
+					if (standardizedTopics.get(i).length()> standardizedTopics.get(j).length()){
+						toDelete.add(standardizedTopics.get(i));
+						// TODO update std_group entry
+						System.out.println("Removing '" + standardizedTopics.get(i) + "' because of '" + standardizedTopics.get(j) );
+						List<ParseTreeChunk> stJ = std_group.get(standardizedTopics.get(j));
+						stJ.addAll(std_group.get(standardizedTopics.get(i)));
+						stJ = new ArrayList<ParseTreeChunk>(new HashSet<ParseTreeChunk>(stJ));
+						std_group.put(standardizedTopics.get(j), stJ);
+					}
+					else {
+						toDelete.add(standardizedTopics.get(j));
+						System.out.println("Removing '" + standardizedTopics.get(j) + "' because of '" + standardizedTopics.get(i) );
+						List<ParseTreeChunk> stI = std_group.get(standardizedTopics.get(i));
+						stI.addAll(std_group.get(standardizedTopics.get(j)));
+						stI = new ArrayList<ParseTreeChunk>(new HashSet<ParseTreeChunk>(stI));
+						std_group.put(standardizedTopics.get(i), stI);
+					}
+
+				}
+			}
+		for(String d: toDelete){
+			//System.out.println("Removed '" + d + "'");
+			standardizedTopics.remove(d);
+		}
+	}
+
+	// substitute synonyms according to internal vocab
+	private String substituteSynonym(String w) {
+		try {
+			for(String[] pair: synonymPairs){
+				if (w.equals(pair[0]))
+					return pair[1];
+			}
+		} catch (Exception e) {
+			e.printStackTrace();
+		}
+		return w;
+	}
+
+	public void generateGroupingReport(String reportName){
+		List<String[]>  report = new ArrayList<String[]>();
+		Set<ParseTreeChunk> chs = entry_group.keySet();
+		report.add(new String[]{"string phrase" , "class", "linguistic phrase",  "list of ling phrases class representatives"});
+
+		for(ParseTreeChunk ch: chs){
+			String head = ch.getLemmas().get(ch.getLemmas().size()-1);
+			List<ParseTreeChunk> values = entry_group.get(ch);
+			if (values.size()<6)
+				head = "";
+			report.add(new String[]{ch.toWordOnlyString(), head,  ch.toString(),  values.toString()});
+		}
+		ProfileReaderWriter.writeReport(report, reportName);
+	}
+
+	//final merge floor-floors-flooring as head nound with phrase update
+	public void applyLastRoundOfAggregation(){
+		//merge <floor - floors - flooring>
+		/*
+			List<ParseTreeChunk> entries =  new ArrayList<ParseTreeChunk>(entry_group.keySet());
+			for(int i=0; i< entries.size(); i++){
+				for(int j=i+1; j< entries.size(); j++){
+					ParseTreeChunk chI = entries.get(i);
+					ParseTreeChunk chJ = entries.get(j);
+					String headI = getLastElement(chI.getLemmas());
+					String headJ = getLastElement(chJ.getLemmas());
+					if (headI==null || headI.length()<MIN_LENGTH_OF_WORD_TO_CONSIDER  || 
+							headJ==null || headJ.length()<MIN_LENGTH_OF_WORD_TO_CONSIDER )
+						continue;
+
+					if (headI.indexOf(headJ)>-1){
+						//leave headJ
+						List<ParseTreeChunk> valuesToAddTo = entry_group.get(chJ);
+						List<ParseTreeChunk> valuesBeingAdded = entry_group.get(chI);
+						if (valuesToAddTo==null || valuesBeingAdded == null)
+							continue;
+						valuesToAddTo.addAll(valuesBeingAdded);
+						entry_group.put(chJ, valuesToAddTo);
+						entry_group.remove(chI);
+						System.out.println("Deleting entry '"+ headI +"' and moving group to entry '"+ headJ +"'");
+					} else if (headJ.indexOf(headI)>-1){
+						//leave headJ
+						List<ParseTreeChunk> valuesToAddTo = entry_group.get(chI);
+						List<ParseTreeChunk> valuesBeingAdded = entry_group.get(chJ);
+						if (valuesToAddTo==null || valuesBeingAdded == null)
+							continue;
+						valuesToAddTo.addAll(valuesBeingAdded);
+						entry_group.put(chI, valuesToAddTo);
+						entry_group.remove(chJ);
+						System.out.println("Deleting entry '"+ headJ +"' and moving group to entry '"+ headI +"'");
+					}
+
+				}
+			}
+		 */
+		for(int i = 0; i<standardizedTopics.size(); i++ )
+			for(int j = i+1; j<standardizedTopics.size(); j++ ){
+				String headI = extractHeadNounFromPhrase(standardizedTopics.get(i));
+				String headJ = extractHeadNounFromPhrase(standardizedTopics.get(j));
+				// if the same word do nothing
+				if (headI.equals(headJ))
+					continue;
+
+				//only if one is sub-word of another
+				if (headI.indexOf(headJ)>-1){
+
+					if (!properSubWordForm(headI, headJ))
+						continue;
+					//entry 'I' will be updated
+					String newKey = standardizedTopics.get(i).replace(headI, headJ);
+
+					List<ParseTreeChunk> stI = std_group.get(standardizedTopics.get(i));
+					List<ParseTreeChunk> stInew = std_group.get(newKey);
+					//if (stInew!=null && !stInew.isEmpty())
+					//	stI.addAll(stInew);
+					if(stI==null)
+						continue;
+					std_group.put(newKey, stI);
+					std_group.remove(standardizedTopics.get(i));
+					System.out.println("Deleted entry for key '"+ standardizedTopics.get(i) +"' and created  '"+ newKey +"'");
+					standardizedTopics.set(i, newKey);
+
+				} else if (headJ.indexOf(headI)>-1){
+					if (!properSubWordForm(headJ, headI))
+						continue;
+					//entry 'J' will be updated
+					String newKey = standardizedTopics.get(j).replace(headJ, headI);
+
+					List<ParseTreeChunk> stJ = std_group.get(standardizedTopics.get(j));
+					List<ParseTreeChunk> stJnew = std_group.get(newKey);
+					//if (stJnew!=null && !stJnew.isEmpty())
+					//	stJ.addAll(stJnew);
+					if(stJ==null)
+						continue;
+					std_group.put(newKey, stJ);
+					std_group.remove(standardizedTopics.get(j));
+					System.out.println("Deleted entry for key '"+ standardizedTopics.get(j) +"' and created  '"+ newKey +"'");
+					standardizedTopics.set(j, newKey);
+				}
+			}
+
+
+
+	}
+
+	private boolean properSubWordForm(String headI, String headJ) {
+		String suffix = headI.replace(headJ, "");
+		if (suffix.equals("s") || suffix.equals("ing") //|| suffix.equals("er") 
+				|| suffix.equals("rooms") ||
+				suffix.equals("") || suffix.equals("counter") ||
+				suffix.equals("room") || suffix.equals("back"))
+			return true;
+
+		//System.out.println("Wrong word '"+ headI + "'reduction into '" + headJ +"'");
+		return false;
+	}
+
+	//generates report 
+	public void generateStdTopicReport(String reportName){
+		List<String[]>  report = new ArrayList<String[]>();
+		report.add(new String[]{"category", "topic", "sub-topics", "phrase instances" });
+
+		for(String t: standardizedTopics){
+
+			String bufCover = "";
+			int count = 0;
+			List<ParseTreeChunk> ptcList = std_group.get(t);
+			if (ptcList == null)
+				continue;
+			for(ParseTreeChunk ch: ptcList){
+				List<String> candidate = TextProcessor.fastTokenize(ch.toWordOnlyString(), false);
+				List<String> tList = TextProcessor.fastTokenize(t, false);
+				List<String> tListChk = new ArrayList<String>(tList);
+
+				tListChk.removeAll(candidate);
+				// fully covered by phrase instance
+				if (!tListChk.isEmpty() || ch.toWordOnlyString().equals(t)){
+					continue;
+				}
+
+				boolean bCovered = true;
+				
+				for(String ts: tList){
+					boolean bCandWordsIsCovered = false;
+					for(String s: candidate){
+						if ((s.indexOf(ts)>-1) )//  && properSubWordForm(s, ts))
+							bCandWordsIsCovered = true;
+					}
+					if (!bCandWordsIsCovered){
+						bCovered = false;
+						break;
+					}
+				}
+				if (!bCovered)
+					continue;
+				bufCover+=ch.toWordOnlyString()+ " # ";
+				count++;
+				if (count > 40)
+					break;
+
+			}
+			if (bufCover.endsWith(" # "))
+				bufCover = bufCover.substring(0, bufCover.length()-3).trim();
+
+			String buf = "";
+			count = 0;
+			// only up to 40 instances of phrases per 1-st level topic
+			for(ParseTreeChunk ch: ptcList){
+				buf+=ch.toWordOnlyString()+ "|";
+				count++;
+				if (count > 40)
+					break;
+			}
+			
+			//TODO uncomment
+			//t = spell.getSpellCheckResult(t);
+			report.add(new String[]{extractHeadNounFromPhrase(t), t, bufCover, buf //, std_group.get(t).toString()
+			});
+		}
+		
+		
+		ProfileReaderWriter.writeReport(report, reportName);
+	}
+	// get a last word from a phrase (supposed to be a head noun)
+	private String extractHeadNounFromPhrase(String topic){
+		String[] tops = topic.split(" ");
+		int len = tops.length;
+		if (len>1){
+			return tops[len-1];
+		}
+		else return topic;
+	}
+
+	// get last elem of a list
+	private String getLastElement(List<String> arrayList ){
+		if (arrayList != null && !arrayList.isEmpty()) {
+			return arrayList.get(arrayList.size()-1);
+		}
+		return null;
+	}
+	/*
+	 * Using Bing API to check if an extracted phrase can be found on the web, therefore is a meaningful phrase 
+	 */
+	public List<String> verifyTopic(){
+		Set<String> phrases = freq.keySet();
+		List<String> approvedPhrases = new ArrayList<String>();
+		for(String p: phrases){
+			List<HitBase> hits = runner.runSearch("\""+p+"\"");
+			for(HitBase h: hits){
+				String lookup = h.getTitle() + " " + h.getAbstractText();
+				if (lookup.indexOf(p)>-1){
+					approvedPhrases.add(p);
+					break;
+				}
+			}
+		}
+		return approvedPhrases;
+	}
+
+	public Set<String> getPhraseLookup(){
+		return freq.keySet();
+	}
+
+	// using phrase frequency to filter phrases
+	public boolean isAcceptablePhrase(String phrase){
+		Integer count = freq.get(phrase.toLowerCase().trim());
+		if (count==null)
+			return false;
+
+		if (count>0 && count < 10000)
+			return true;
+		return false;
+	}
+
+	public static void main(String[] args){
+		LinguisticPhraseManager man = new  LinguisticPhraseManager(
+				"/Users/bgalitsky/Documents/workspace/move_com/phrasesOfInterest.csv");
+		man.doLingGrouping();
+		man.generateGroupingReport("topics_groups7_mergedHeads.csv");
+		List<String> stdTopics = man.formStandardizedTopic();
+		man.applyLastRoundOfAggregation();
+		man.generateStdTopicReport("std_topics7_mergedHeads.csv");
+		System.out.println(stdTopics);
+
+	}
+}


[22/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/105TeleHTC_Manual_11_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/105TeleHTC_Manual_11_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/105TeleHTC_Manual_11_EN.txt.txt
new file mode 100644
index 0000000..14d38d2
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/105TeleHTC_Manual_11_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Chapter 11 Camera and Multimedia 11.1 Taking Photos and Videos Use Camera to take photos and record video clips . To open the Camera screen On the Home screen , slide to the Photos and Videos tab and then tap the camera icon ( ) . The screen orientation automatically switches to landscape mode when you open Camera . To close the Camera Tap the On-screen control button ( ) , and then tap . Changing the capture mode Tap , then tap the capture mode icon ( \u2014 the icon displayed is based on the current capture mode ) . On the menu , tap the capture mode you want to use . Icon Capture Mode Captures standard still images . Records video clips , with or without audio . 196 Camera and Multimedia Icon Capture Mode Captures still images continuously in one direction , and then stitches them to create a panoramic view of a scenery . Records video clips suitable for sending via MMS . Captures a still image that you can assign as a contact picture . Captures still images and places them within f
 rames . Note When using the second camera to capture , you can use all modes , except for Panorama . Using the focus Note Use the automatic focus or Touch Focus function to capture sharp and crisp photos and videos . The second ( front ) camera has no auto-focus function . Auto-focus Auto-focus sets the focus on the object at the center of the screen and is available for all capture modes . By default , touching the Virtual Camera button ( ) activates auto-focus and a flashing white bracket ( ) appears in the center of the screen . When focus is set , the bracket changes to a steady green bracket ( ) , and then the Camera takes the photo or starts to record the video . Touch Focus Touch Focus allows you to focus on a particular area of the screen . Touch Focus is enabled by default and is only available for Photo or Contacts Picture mode . Note To disable Touch Focus , see \u201c Setting advanced options \u201d in this chapter . Camera and Multimedia 197 To use Touch Focus : 1. Set the Ca
 mera to Photo or Contacts Picture mode . 2. Touch the area on the screen that you want to focus . The Touch Focus indicator ( ) positions itself on the selected area . Note If you tap outside the effective focus area , a white box appears to indicate the area where you can use Touch Focus . 3. Touch to activate auto-focus on the selected area and take the photo . See \u201c Auto-focus \u201d in this chapter for more information . Tip To move the Touch Focus indicator back to the center of the screen , tap . Capturing photo or recording video By default , the Camera automatically takes the shot or starts capturing video after you have set the focus . \u2022 In Photo or Contacts Picture mode , touch the Virtual Camera button to activate auto-focus ; when focus is set , the Camera automatically takes the shot . \u2022 In Video or MMS Video mode , touch the Virtual Camera button to activate auto-focus ; when focus is set , the Camera automatically starts recording . Touch the Virtual camera button 
 again to stop recording . \u2022 In Picture Theme or Panorama mode , touch the Virtual Camera button to activate auto-focus ; when focus is set , the Camera automatically takes the first shot . Do the same for each shot you need to take . 198 Camera and Multimedia The Review screen Icon After capturing a photo or recording video , the Review screen gives you the option to view/play , send , or delete the photo or video clip . Function Return to the live Camera screen . / View the captured image or play the captured video in Album . See \u201c Viewing Photos and Videos Using Album \u201d in this chapter . Delete the captured image or video . Send the captured image or video via MMS or e-mail . After capturing in Contacts Picture mode , tap to assign the photo to a contact . On-screen controls Tap to display the on-screen controls . 1 2 3 4 5 6 7 1 Album . Open the Album program . See \u201c Viewing Photos and Videos Using Album \u201d for details . 2 Capture mode . See \u201c Changing the capture mode
  \u201d for details . 3 Touch Focus / Auto-focus indicator . See \u201c Using the focus \u201d for details . Camera and Multimedia 199 4 Menu . Open the Menu Panel . See \u201c Menu Panel \u201d for details . 5 Virtual Camera button . Activates the auto-focus and functions as the shutter release button . 6 Template Selector icon . In Picture Theme mode , tap to select another frame . 7 Progress indicator . In Picture Theme and Panorama modes , this shows the number of consecutive shots . Menu Panel The Menu panel lets you quickly adjust common camera settings and provides access to more advanced camera settings . To open the Menu panel , tap on the Camera screen , and then tap . To adjust a setting , keep tapping a button , for instance , White Balance , to cycle through the available settings . Note The available menu panel buttons depend on the selected capture mode . Icon Function Select the white balance setting according to the current lighting condition to capture colors more accurately . Op
 en the Brightness bar and adjust the brightness level . Tap outside the Brightness bar to apply the change . Select the camera to use . You can choose from main camera ( second camera ( ) . ) or Set the ISO level manually to 100 , 200 , 400 , or 800 , or set it back to AUTO . Higher ISO numbers are better for taking pictures in low light conditions . Note The camera automatically adjusts the ISO level up to 400 only . Set the self-timer to 2 seconds , 10 seconds , or Off when in Photo or Contacts Picture mode . Open the advanced camera settings . See \u201c Setting advanced options \u201d for details . 200 Camera and Multimedia Zooming Before capturing a photo or a video clip , you can zoom in or out on your subject . To zoom in or out , slide your finger up or down on the Zoom bar while holding the device sideways . While you are sliding your finger on the Zoom bar , the level of zoom is shown on the zoom control bar on the screen . Zoom control bar Zoom bar The camera zoom range for a p
 hoto or a video clip depends on the capture mode and resolution . Refer to the table for more information . Capture mode Photo Resolution Normal : 2048 x 1536 3M Widescreen : 2048 x 1216 Normal : 1600 x 1200 2M Widescreen : 1600 x 960 Normal : 1280 x 960 1M Widescreen : 1280 x 768 Normal : 640 x 480 L Widescreen : 640 x 384 Normal : 320 x 240 M Widescreen : 320 x 192 Zoom range 1.0x to 1.6x 1.0x to 1.6x 1.0x to 1.6x 1.0x to 1.6x 1.0x to 3.2x Camera and Multimedia 201 Capture mode Video Resolution VGA 640 x 480* CIF 352 x 288 L 320 x 240 M 176 x 144 S 128 x 96 M 176 x 144 S 128 x 96 L 640 x 480 M 320 x 240 M 272 x 272 Determined by the current template Zoom range 1.0x to 1.6x 1.0x to 1.6x 1.0x to 1.6x 1.0x to 1.6x 1.0x to 1.6x 1.0x to 1.6x 1.0x to 1.6x 1.0x to 1.6x 1.0x to 3.2x 1.0x to 3.2x Depends on the size of the selected template MMS Video Panorama Contacts Picture Picture Theme * Available only in some models . Note The capture resolutions and zoom ranges are only for the main 
 camera . Setting advanced options The Advanced options let you adjust options for your selected capture mode and set Camera program preferences . Tap Advanced on the Menu panel to open the Advanced screen . Tap an item to change the setting . Tap to go back to the Camera screen . Tap to navigate through the Advanced option screens . 202 Camera and Multimedia \u2022 Resolution . Select the resolution to use . \u2022 Widescreen ( Photo mode ) . When On ( default ) you can use the entire screen as viewfinder . Refer to the zoom table for the resolution when Widescreen is On . \u2022 Touch Focus ( Photo and Contacts Picture modes ) . Enables or disables the Touch Focus function . Touch Focus is On by default . \u2022 Review Duration . Set the time for a captured photo/video to remain on the Review screen . \u2022 Quality . Select the image quality level for captured photos . \u2022 Capture Format ( Video and MMS Video modes ) . Select the file format for recorded videos . \u2022 Time Stamp ( Photo mode ) . 
 Include the date and time on captured photos . \u2022 Storage . Select where to save photos and videos . You can either save the files to the main memory , or to a storage card if available . \u2022 Keep Backlight . Turn the backlight on or off . This overrides your device backlight settings while you are using the camera . \u2022 Shutter Sound . Play a shutter sound when you tap the Virtual Camera button . \u2022 Image Properties . Adjust the image properties . 1 2 Tap a property to adjust . Save the settings . Decrease the value . Reset all properties to default . Increase the value . Close the Image Properties screen without applying and saving the changes . 1 3 4 5 6 2 3 4 5 6 \u2022 Effect . Apply a special effect to your photos or videos . Camera and Multimedia 203 \u2022 Metering Mode . Select a metering mode to allow the camera to measure the amount of light and calculate the best-fit exposure value before capturing . \u2022 Prefix . The file name of a newly captured photo or video is set to \u201c
  IMAGE \u201d or \u201c VIDEO \u201d followed by a sequential number , for example : IMAGE_001 . jpg . You can also choose the current date or date and time as the file name prefix . Note You can not change the file name prefix when saving captured photos to a storage card . Captured photos are named using the convention IMAGnnnn . jpg ( where \u2018 nnnn \u2019 is the counter ) , and are saved to the \DCIM\100MEDIA folder on the storage card . \u2022 Counter . Reset the file naming counter back to 1. \u2022 Flicker Adjustment . When taking indoor shots under fluorescent lighting , keep this setting on Auto or change it to the proper frequency ( 50Hz or 60Hz ) of the power in your country to reduce flicker . \u2022 Grid ( Photo mode ) . Show a grid on the screen so you can easily frame and center your subjects . \u2022 Record with Audio ( Video or MMS Video mode ) . Select On to record videos with audio , or Off to record videos without audio . \u2022 Recording Limit ( Video mode ) . Set the maximum duration or 
 file size for recording video . \u2022 Template ( Picture Theme mode ) . Select a template . \u2022 Template Folder ( Picture Theme mode ) . By default , templates are stored in the \My Documents\Templates folder on the device\u2019s Main Memory . If you transferred some templates to a storage card , set this option to Main + Card . \u2022 Show Reminder ( Contacts Picture mode ) . Select On if you want the Camera to prompt you to assign the captured picture to a contact . \u2022 Direction ( Panorama mode ) . Choose the direction in which to stitch the photos in a panorama . \u2022 Stitch Count ( Panorama mode ) . Select the number of shots to take and stitch into a panorama . 204 Camera and Multimedia 11.2 Viewing Photos and Videos Using Album Use Album to view photos and videos in your device . To open Album On the Home screen , slide to the Photos and Videos tab , and then tap Album . You can also tap while on the Camera program to open Album . Open the Camera program . View the photo in full scree
 n . View another album . Play the video file . Start a slideshow , set the album as favorite , and more . Camera and Multimedia 205 Viewing photos While viewing a photo , tap the screen to open the menu . Send the photo using e-mail or MMS . Save the photo as a contact picture , save the on-screen image as a new file , and more . Start the slideshow . Return to the previous screen . To view the next or previous photo in the album When you are holding your device upright , view the next or previous photo by swiping your finger up or down the screen . When you turn your device sideways , view the next or previous photo by swiping your finger left or right across the screen . To zoom in or out on an photo Double-tap the area on the photo you want to zoom in on . To zoom out , doubletap the screen again . Tips \u2022 Use the zoom bar to zoom in on the center of the photo . Slide your finger to the right of the zoom bar to zoom in . Slide to the left of the zoom bar to zoom out . \u2022 When y
 ou zoom in on a photo , tap and hold the screen to pan and view other areas of the photo . When you zoom in on a photo , tap the screen , tap and then select Save Screen Image to save the zoomed in photo as a new file . 206 Camera and Multimedia To view a slideshow of photos in the current album While viewing a photo , tap the screen , and then tap Note . Animated GIF and video files are excluded from the slideshow . During the slideshow , tap the screen to show the playback controls . Return to the previous screen . Set the transition effect , time per slide , and orientation of the photos . Go back to the previous image . Tip Go to the next image . Play or pause the slideshow . If the photos are in your favorite album , you can start the slideshow from the Photos and Videos tab on the Home screen . See \u201c Photos and Videos \u201d in Chapter 3 for details . Camera and Multimedia 207 Playing videos While playing back video , tap the screen to show the playback controls . Drag to go fo
 rward or backward in the video . Elapsed time Go back to the beginning of the video . Play or pause Stop the video . the video . Total duration Change the video display mode to Fit & Crop ( default ) , Best Fit , or Stretch . Closing Album Tap When viewing photos in full screen , tap the screen to show the on-screen menu or playback controls and then tap to go back to the main Album screen . on the main Album screen to close Album . 208 Camera and Multimedia 11.3 Using Windows Media� Player Mobile Windows Media� Player Mobile plays digital audio and video files that are stored on your device or on the web . To open Windows Media� Player Mobile Tap Start > All Programs > Windows Media . Windows Media� Player Mobile opens with the Library screen . Use the Library screen to find and play songs , videos , and playlists that are stored on your device or on the storage card . Playing media To update the Library In most cases , Windows Media� Player Mobile automatically updates the li
 brary . However , you can manually update the library to ensure that it contains new files that you recently copied to your device or to the storage card . Tap Menu > Update Library to manually update the library list . To play audio and video files 1. On the Library screen , tap the Library arrow ( near the top of the screen ) , then tap the media storage that you want use , for example , Storage Card . 2. Select a category ( for example , My Music or My Playlists ) . 3. Tap and hold the item that you want to play ( such as a song , album , or artist name ) , and then tap Play . Camera and Multimedia 209 The playback screen opens and plays the audio or video file . Drag to jump to any part of the video or audio . Full screen Mute/Unmute Increase volume Visit WindowsMedia . com Decrease volume Skip to the beginning of Skip to the next file . the current file or to the Play/Pause previous file . Synchronizing video and audio files Use the latest version of Windows Media� Player on y
 our computer to synchronize digital media files from your computer to your device . This ensures that protected files and album art ( for skins ) are correctly copied to your device . To automatically synchronize content to your device 1. On your computer , open Windows Media� Player , then connect your device to your computer . 2. In the Device Setup Wizard , select Yes , search my computer now . 210 Camera and Multimedia Note If you have previously opened Windows Media� Player and searched for media on your computer , you are not prompted to search your computer in step 2. 3. On the Device Setup dialog box , enter a name for your device and click Finish . Note If you installed a storage card that has a capacity of more than 4GB and your entire library can fit into the storage card , Windows Media Player will automatically synchronize your music files . The device also needs to be set to Disk Drive mode for Windows Media Player to automatically synchronize . 4. On the left panel 
 of Windows Media� Player , right-click the name of your device and click Set Up Sync . Note To set up media synchronization on a storage card , right-click Storage Card in the left panel of Windows Media� Player , then click Set Up Sync . 5. Select the playlist(s ) that you want to sync between your computer and device , then click Add . Note On the Device Setup dialog box , make sure that the Sync this device automatically check box is selected . 6. Click Finish . The files begin synchronizing to your device . The next time you connect your device to your computer while Windows Media� Player is running , synchronization will start automatically . To manually synchronize content to your device 1. If you have not set up media synchronization between your device and computer , follow steps 1 to 3 in \u201c To automatically synchronize content to your device \u201d . 2. Click the Sync tab on the Windows Media� Player of your computer . Select a Playlist or a Library on the left panel of 
 the Windows Media� Player . 3. From the Content List , drag the media files that you want to sync to your device and drop them to the Sync List . Camera and Multimedia 211 Playlists and Library List Content List Sync List 4. Click Start Sync to start synchronizing the selected files to your device . Notes \u2022 Use Windows Media� Player 11 or higher on your computer to synchronize media files to your device . \u2022 Audio files copy faster if Windows Media� Player is configured to automatically set the quality level for audio files copied to your device . For more information , see the Windows Media� Player Help . Creating playlists Create a new playlist by saving the current Now Playing playlist and giving it a new name . To save a new playlist 1. If you are not on the Library screen , tap Menu > Library . 2. Tap to select a category ( for example , My Music or My Playlists ) . 212 Camera and Multimedia 3. Select a media file that you want , then tap Menu > Queue Up . This adds the 
 file to the Now Playing list . Repeat this step until you have added all desired media files to the Now Playing list . Note You cannot select multiple files simultaneously . 4. After adding the media files , tap Menu > Now Playing . 5. On the Now Playing screen , tap Menu > Save Playlist . 6. Enter the playlist name , then tap Done . 7. To play back the playlist you created , tap My Playlists in the Library , select your playlist , then tap Play . Troubleshooting For troubleshooting information , visit http://www.microsoft.com/windowsmobile/ en-us/help/more/windows-media-player . mspx Camera and Multimedia 213 11.4 Using Audio Booster Audio Booster optimizes sound by providing an audio equalizer for a better listening experience . To use Audio Booster , connect the supplied wired headset to your device . To open Audio Booster Tap Start > All Programs > Audio Booster . 1 3 1 2 3 4 Enable or disable the equalizer . Apply the modified settings . Choose the equalizer preset to use . Dra
 g the sliders to adjust the frequency . Exit Audio Booster without applying the modified settings , or save or delete preset equalizer settings . 4 5 2 5 To create a custom equalizer preset 1. Adjust the frequency bands to your desired values by dragging the equalizer controls . 2. When finished , tap Menu > Save as Preset . 3. Enter a preset name and then tap Done . The equalizer preset is added to the preset list box . Note To delete a custom equalizer preset , select it on the preset list box and then tap Menu > Delete Preset . You cannot delete pre-installed equalizer presets . 214 Camera and Multimedia 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/106TeleHTC_Manual_12_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/106TeleHTC_Manual_12_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/106TeleHTC_Manual_12_EN.txt.txt
new file mode 100644
index 0000000..8c65a50
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/106TeleHTC_Manual_12_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Chapter 12 Programs 12.1 Managing Programs Launching a program Tap Start > All Programs , and then tap a desired program . To view more programs , scroll down the screen by swiping your finger upward . See \u201c Start Screen \u201d in Chapter 1 for details . Installing a program Note You can install or uninstall additional programs . Your device does not allow you to uninstall most of the preinstalled programs . Before you purchase additional programs , make sure that they are compatible with your device . Before you launch the program installer , check first whether the installer can directly run on Windows Mobile or needs to be run on a computer . To install a program directly on your device 1. If the installer is a . cab file , directly download or copy it to your device . 2. On your device , open File Explorer and navigate to the installer file . 3. Tap the file to launch the installer . 216 Programs 4. Choose whether to install the program into the main memory or your storage card ,
  if you have the storage card inserted on your device . 5. Wait for the installation to complete . Note You may be prompted to restart your device after installation is completed . To install a program from your computer 1. If the installer is an . exe file such as Setup . exe , most likely , the installer needs to be run on a computer . Download or copy it to your computer . 2. Connect your device to your computer using the USB cable , and then doubleclick the installer file on your computer to launch the installer . 3. Follow the on-screen instructions on your computer and your device to complete the installation . To uninstall a program 1. Tap Start > Settings , and then tap All Settings . 2. On the System tab , tap Remove Programs . 3. In the Programs in storage memory list , select the program you want to remove , and then tap Remove . Programs 217 12.2 Contacts Contacts is your address book for people and businesses you communicate with . You can add Outlook contacts from your
  device or synchronized with your PC or the Exchange Server , SIM contacts stored on the SIM card , or Windows Live\u2122 contacts . Note You can only add Windows Live\u2122 contacts after you have set up Windows Live\u2122 on your device . See \u201c Using Windows Live\u2122 \u201d in Chapter 8. To open the Contacts screen Tap Start > All Programs > Contacts . Adding new contacts To add a contact on your device 1. On the Contacts screen , tap New , select Outlook Contact , and enter the contact information . 2. In the File as field , choose how you want the contact name to appear in the contact list . 3. To add a photo of the contact , tap Select a picture , and then select the picture file or tap Camera to take a photo of the contact . 4. To assign a ring tone to the contact , tap the Ring tone field , scroll through the list of tones , then select a desired tone . 5. When finished , tap OK . Tip To edit contact information , tap the contact name and then tap Menu > Edit . To add a contact to your 
 SIM card 1. On the Contacts screen , tap New , then select SIM Contact . 2. Enter a contact name and phone number , and then tap OK . Tip You can also use the SIM Manager to add and edit contacts on your SIM card . For more information , see \u201c SIM Manager \u201d in this chapter . 218 Programs Organizing and searching contacts To group similar contacts For easier contact management , you can group contacts by assigning them into categories . 1. Create a new Outlook contact or edit an existing Outlook contact . 2. Tap Categories . 3. Select a preset category such as Business or Personal , or tap New to create your own category . 4. When finished , tap OK . To find a contact on your device There are several ways to find a contact when your contact list is long : \u2022 \u2022 Begin entering a contact name in the provided text box until the contact you want is displayed . On the Quick Scroll bar on the right of the Contacts screen , scroll to or tap a letter to go directly to the part of the c
 ontacts list where contact names start with that letter . Note You can disable the Quick Scroll bar by tapping Menu > Options and selecting the Show alphabetical index check box . This shows the alphabetical index at the top of the Contacts list . Quick Scroll bar \u2022 Filter the list by categories . In the contact list , tap Menu > Filter , then tap a category you\u2019ve assigned to a contact . To show all contacts again , select Menu > Filter > All Contacts . Programs 219 Sharing contact information You can quickly send contact information to another mobile phone or device through Bluetooth beaming or text messaging . To send contact information via Bluetooth 1. Tap and hold a contact on the Contacts screen . 2. Tap Send via Bluetooth , then select the device where to send the contact . Note Before sending , make sure Bluetooth is turned on and set to visible mode on your device and the target mobile device . You can also send contact information to your computer . See Chapter 9 for 
 details . To send contact information via text messaging 1. Tap and hold a contact on the Contacts screen . 2. Tap Send Contact > SMS / MMS . 3. Select the contact information you want to send , and then tap Done . 4. In the new text message , enter the mobile phone number of the recipient , and then tap Send . 220 Programs 12.3 JBlend Use JBlend to download , install and manage MIDlets or MIDlet suites on your device . MIDlets are Java applications such as games and tools that can run on mobile devices , while a MIDlet suite is a collection of one or more MIDlets . Your device supports Java 2 Micro Edition ( J2ME ) . To open JBlend Tap Start > All Programs > JBlend . Installing and launching MIDlets/MIDlet suites To download and install from the Internet 1. Open an Internet browser , such as Opera Mobile . 2. Navigate to the web page with the link for the MIDlet/MIDlet suite you want to download , and tap the link . 3. Follow the onscreen instructions to install and run the MIDlet/
 MIDlet suite . To install from your device You can copy MIDlets/MIDlet suites from your PC to your device using the USB sync cable or Bluetooth connection . 1. On the JBlend screen , tap Install > Local Install . The program then searches for MIDlets/MIDlet suites on your device and displays them in a list . 2. From the list , tap the MIDlet/MIDlet suite that you want to install . 3. Follow the onscreen instructions to install and run the MIDlet/MIDlet suite . To run a MIDlet/MIDlet suite On the JBlend screen , open the folder that contains the MIDlet/MIDlet suite , and then tap the MIDlet/MIDlet suite . Programs 221 Managing MIDlets/MIDlet suites You can update MIDlets/MIDlet suites , or uninstall them to free up memory space . To update a MIDlet/MIDlet suite 1. On the JBlend screen , tap the Input Panel icon , and then tap the up/down navigation keys on the on-screen keyboard to select the MIDlet/MIDlet suite that you want to update . 2. Tap Menu > Application > Update Info . JBle
 nd then searches for updates for the selected MIDlet/MIDlet suite . To uninstall a MIDlet/MIDlet suite Before you uninstall a MIDlet , make sure that it is not running . 1. On the JBlend screen , tap and hold the MIDlet , and then tap Delete . 2. Tap Yes to confirm . Note To uninstall all MIDlets and folders , tap Menu > Application > Delete All . 222 Programs 12.4 JETCET Presentation If you are away from your office and urgently need to make a presentation , you can use JETCET Presentation to create a PowerPoint file and present it from your device . JETCET Presentation also allows you to open and display presentations that were created in Microsoft PowerPoint 97 , PowerPoint 2000 , PowerPoint 2002 ( XP ) , or PowerPoint 2003. Opening JETCET Presentation Creating a new presentation Tap Start > All Programs > JETCET Presenter . When you open JETCET Presentation , the main screen displays a folder tree . Tap Menu > Cancel first and then tap Menu > File > New to create a new presentat
 ion . You can then enter text , insert graphics , and add more slides . To add text to a slide 1. Tap Edit > Shape > Insert , tap , and then tap OK . 2. Drag your finger or stylus to draw a text box on the slide . 3. While the text box is selected , set the font type , size and style first and then start entering your text using the hardware keyboard or on-screen keyboard . 4. While the text box is selected , tap Menu > Paragraph and then set the paragraph alignment , indentation , and line spacing . Tip To format the text with automated bullets or numbering , tap Menu > Paragraph > Bullets . 5. To modify the properties of the text box , select the text box and then tap Edit > Shape > Modify . You can change the fill color , add or remove the border , and more . To wrap text inside the text box , tap the Text Box tab and then select the Word wrap text in AutoShape check box . Programs 223 To edit text 1. Select the text box that contains the text you want to edit . 2. Tap Edit > Tex
 t > Modify and then change your text . Tap within the text region to place the cursor exactly where you want to start editing . To insert an object shape 1. Tap Edit > Shape > Insert . 2. Tap an object , for example , an arrow , from the Autoshape Objects . 3. Drag on the slide to draw the select shape . Tip To insert a picture , tap Edit > Shape > Picture . To add and edit slides 1. Tap Edit > Slide > New to add a new slide . 2. To change the background color of a slide , tap Edit > Slide > Background . 3. In Background fill , tap Fill Effects to apply a gradient or tap More Colors to choose a solid color . 4. Tap Apply to change the background color of the current slide , or tap Apply to All to change all slides . 5. To apply a transition effect to the slides , tap Edit > Slide > Transition . 6. In Effect , select the type of transition effect to use and then select a speed . 7. Choose whether to advance slides on mouse click or automatically . If you select to auto-display slides
  , set the time delay . 8. To make the changes to the current slide only , tap Apply . For the changes to affect all slides in the presentation , tap Apply to All . To save the presentation If this is your first time to save your presentation , tap Menu > File > Save As and then enter a new file name . To save changes made to your presentation , tap Menu > File > Save . 224 Programs Displaying the presentation You can connect your device to a large-screen display such as a television and show your presentation to an audience as a full-screen slideshow . For more information about connecting your device to a large-screen display , see \u201c Connecting to an External Display \u201c in Chapter 13. To display the presentation as a slideshow 1. Tap the PowerPoint file in the folder tree to open the file , or tap File > Open and then tap the PowerPoint file . 2. Tap Menu > Show > Set Up Show . Select whether to advance slides Manually or Using timings , if present , and then tap OK . 3. Tap Me
 nu > Show > View Show . The presentation is then displayed in landscape and in full screen . 4. Slide open the hardware keyboard . If you have set up the slideshow to advance manually , press the right arrow key to advance one slide forward and the left arrow key to go back one slide . Another way to advance one slide forward is to tap the screen . You can also tap the arrow watermark at the lower-left corner of the screen to open a selection menu , then tap Next or Previous to move between slides . Arrow watermark Programs 225 To exit the slideshow Tap the arrow watermark , tap the down arrow to scroll down the selection menu , and then tap End Show . For more information about using JETCET Presentation , tap JETCET Presentation to view the program help . on the toolbar of 12.5 Microsoft� Office Mobile Microsoft� Office Mobile consists of the following applications : \u2022 Microsoft� Office Excel� Mobile lets you create and edit Excel workbooks and templates on your device . \u2022 
 Microsoft� Office OneNote� Mobile lets you create notes with text , photos , and voice recordings for synchronization later with Microsoft� Office OneNote� 2007 on your computer . \u2022 Microsoft� Office PowerPoint� Mobile allows you to view ( not create ) slide show presentations in * . ppt and * . pps formats . \u2022 Microsoft� Office Word Mobile lets you create and edit documents and templates in Word Mobile and save them as * . doc , * . rtf , * . txt , and * . dot files . To use Microsoft Office Mobile Tap Start > All Programs , and then tap the Office Mobile application that you want to use . Word Mobile and Excel Mobile do not fully support some features of Microsoft� Office Word and Microsoft� Office Excel� . To find out which features are not supported , tap Start > All Programs > Help and then search for \u201c unsupported \u201d in the Help . 226 Programs Tips \u2022 By default , Word Mobile saves documents in . docx format , while Excel Mobile saves workbooks in . xlsx form
 at . If you want to open these types of files on a computer that uses Microsoft Office 2000 , Office XP , or Office 2003 , you need to download and install the File Format Converter in your computer . Download the File Format Converter at http://www.microsoft.com/downloads/. \u2022 If you want Word Mobile to save documents in . doc format , tap Menu> Tools > Options ( or Menu > Options when no document is opened ) , then change the Default template to Word 97-2003 Document ( . doc ) . \u2022 If you want Excel Mobile to save workbooks in a format that is compatible with Microsoft Office 2000 , Office XP , or Office 2003 , tap Menu> Options ( with no workbook opened ) , then change the Template for new workbook to Blank 97-2003 Workbook . 12.6 SIM Manager SIM Manager allows you to view the contents of your SIM card , transfer contacts from your SIM card to your device or vice versa , and make a call to a SIM contact . To create a contact on your SIM card 1. Tap Start > All Programs > SIM Ma
 nager . 2. Tap New and enter the contact name and phone number . 3. Tap Save . Tip To change contact information on your SIM card , tap a SIM contact , edit the information , and then tap Save . To copy SIM contacts to your device If you have saved contacts on your SIM card , you can copy them into Contacts on your device . 1. On the SIM Manager screen , select the desired contact , or select all SIM contacts by tapping Menu > Select All . 2. Tap Menu > Save to Contacts . Programs 227 To copy contacts to the SIM card Only one phone number per contact name can be stored on a SIM card . 1. On the SIM Manager screen , tap Menu > Contacts to SIM . 2. Select the check boxes of the contact\u2019s phone numbers that you want to copy to your SIM card , then tap Save . When you copy a contact that has several phone numbers to your SIM card , SIM Manager separately saves each number by appending an indicator at the end of each name . By default , /M , /W , and /H are appended to indicate mobile 
 , work , and home phone numbers respectively . To edit the default indicators and to select which other types of numbers to save to the SIM card , tap Menu > Tools > Options . 12.7 WorldCard Mobile WorldCard Mobile lets you take a photo of a business card and easily export the contact information on the card into your Contacts . To capture a business card and export its information to Contacts 1. Tap Start > All Programs > WorldCard Mobile . 2. Tap on the WorldCard Mobile screen to open the Camera program . 3. Tap the Virtual Camera button ( ) to take a photo of the business card , then tap the Back button ( ) on the Review screen . Refer to Chapter 11 for details on how to use the camera . Tip Make sure that the business card is displayed completely on the Camera screen before you take the photo , and that lighting is adequate . 4. On the WorldCard Mobile screen , tap Recognize . Tips \u2022 If you want to retake the shot , tap Re-Capture . \u2022 For better recognition , choose the lang
 uage and region based on the language used in the business card . 228 Programs 5. The screen shows various contact information from the business card . Edit the contact information , if needed . Clear the check boxes of items that you do not want to be exported . 6. Tap Export , then go to your Contacts list to view the exported contact . For more information about using WorldCard Mobile , tap Mobile screen to view the program help . on the WorldCard 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/107TeleHTC_Manual_13_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/107TeleHTC_Manual_13_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/107TeleHTC_Manual_13_EN.txt.txt
new file mode 100644
index 0000000..50327b2
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/107TeleHTC_Manual_13_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Chapter 13 Managing Your Device 13.1 Copying and Managing Files Copy files between your device and computer , or copy files to a storage card installed on your device . You can also efficiently manage your files and folders using File Explorer . To copy files using Windows Mobile Device Center or ActiveSync 1. Connect your device to the computer . 2. Click File Management > Browse the contents of your device in Windows Mobile Device Center on your computer , or click Explore in ActiveSync on your computer . This displays the content of your device in Windows Explorer on your computer . 3. To copy a file from your device to your computer : a. Browse the contents of your device , right-click the file you want to copy , and then click Copy . b. Right-click a folder on your computer , and then click Paste . 4. To copy a file from your computer to your device : a. Browse the folders on your computer , right-click the file you want to copy , and then click Copy . b. Right-click a folder o
 n your device , and then click Paste . 230 Managing Your Device To manage files on your device using File Explorer File Explorer lets you browse and manage the contents of your device . The root folder on the device is named My Device , and contains the following folders : My Documents , Program Files , Windows , and more . 1. Tap Start > All Programs > File Explorer . 2. Tap a folder or file to open it . 3. To return to an upper level folder , tap Up . 4. To quickly delete , rename , or copy a file , tap and hold the file , and then choose the desired operation on the shortcut menu . To copy or delete multiple files , tap and drag over the desired files , tap and hold the selection , and then choose an option from the menu . To copy files to a storage card 1. Make sure that the storage card is properly installed on your device . 2. Connect the device to your computer using the USB cable , select Disk Drive on the Connect to PC screen , and then tap Done . 3. On your computer , navi
 gate to the removable disk drive , and then start copying files to the storage card . 4. When finished , disconnect the device from your computer . Managing Your Device 231 13.2 Settings on your Device You can adjust the device settings to suit the way you work . Use the Settings tab to adjust basic device settings . See \u201c Settings \u201d in Chapter 3 for details . To see all available settings , tap Start > Settings , and then tap All Settings . Personal tab Icon Description Buttons Assign a program or function to a hardware button or set the function you want to execute ( for instance , lock the device ) when you press and hold the END button . Input Set options for each of the input methods . For instance , you can set the device to play a sound or vibrate when you tap a Full QWERTY or Phone Keypad key . Keyboard Sliding Sound Select a sound to play when you open and close the hardware keyboard . Lock Set a password for your device . Menus Set which programs appear in the Start me
 nu . Note The Start menu is disabled when the TouchFLO 3D Home screen is enabled . To enable the Start menu , tap Start > Settings , tap All Settings > Personal tab > Today > Items tab , and then clear the TouchFLO 3D check box . Owner Information Enter your personal information on your device . Phone Customize phone settings such as the ring tone , or set a PIN number for your SIM card . 232 Managing Your Device Icon Description Sounds & Notifications Enable sounds for events , notifications , and more , and set the type of notification for different events . Today Customize the appearance of the Home screen and the information to display on it . Note You will not see the Today screen when the TouchFLO 3D Home screen is enabled . To display the Today screen , tap Start > Settings button , tap All Settings > Personal tab > Today > Items tab , and then clear the TouchFLO 3D check box . System tab Icon Description About This is where you can see basic information such as the Windows M
 obile� version and type of processor used on your device . You can also specify a name for your device . Auto Volume Control Automatically adjust the recording volume to an even level . Certificates See information about certificates that are installed on your device . Clear Storage Reset your device to factory default settings . Clock & Alarms Set the device clock to the date and time of your locale or to a time zone you are visiting . Alarms can also be set on specified days and times of the week . Customer Feedback Set your device to automatically send information about how you use Windows Mobile . By default , this option is disabled . Device Information See information about the firmware version , hardware , identity , and the duration of calls made on your device . Managing Your Device 233 Icon Description Encryption Encrypt files on your storage card . Encrypted files are only readable on your device . Error Reporting Enable or disable the device 's error reporting function 
 . When this function is enabled and a program error occurs , technical data about the state of the program and your computer is logged in a text file and delivered to Microsoft 's technical support if you choose to send it . External GPS Set the appropriate GPS communication ports , if required . You may need to do this when there are programs on your device that access GPS data or you have connected a GPS receiver to your device . G-Sensor Recalibrate the G-Sensor of the device . Use this function when auto-rotate is not working properly . Managed Programs Displays the programs that were installed on your device using System Center Mobile Device Manager . Memory Check the device memory allocation and storage card information . Power Check the remaining battery power . You can also set the brightness of the backlight and the timeout for turning off the display ( switching the device to Sleep mode ) to conserve battery power . Regional Settings Set the regional configuration to use ,
  including the format for displaying numbers , currency , date , and time on your device . Remove Programs Uninstall programs that you previously installed on your device . Screen Recalibrate the screen , apply ClearType to display text more clearly , and change the screen text size . Task Manager Stop running programs and set the ( ) button to end programs immediately when you tap the button . For more information , see \u201c Using Task Manager \u201d in this chapter . 234 Managing Your Device Icon Description TouchFLO Set the device to vibrate when you tap the tabs , links , or other onscreen items of supported applications . TV Out Set the video standard to use when outputting to an external display . Windows Update Link to Microsoft 's web site and update Windows Mobile� on your device with the latest security patches or fixes , if they are available . Connections tab Icon Description Advanced Network Turn HSPA on/off , set the GPRS authentication method , and set the line type to u
 se when making a circuit switch data ( CSD ) connection . HSPA is a 3.5G digital data service that provides high-speed downstream/upstream data rates . Check with your wireless service provider whether HSPA is supported in your area . Bluetooth Turn Bluetooth on , set your device to visible mode , and scan for other Bluetooth devices . Comm Manager Manage your device\u2019s connections ( including the phone , GPRS/3G , Wi-Fi , and Bluetooth ) or turn the phone function on or off . Connection Setup Configure your device\u2019s data connections , such as GPRS , and WAP , based on the detected SIM card or selected network operator . See \u201c Using Connection Setup \u201d in this chapter for details . Connections Set up one or more types of modem connection for your device , such as phone dial-up and GPRS , so that your device can connect to the Internet or a private local network . Domain Enroll Enroll your device in your company\u2019s domain to let System Center Mobile Device Manager manage the d
 evice . Ask your network administrator for more details . Managing Your Device 235 Icon Description USB to PC Set the type of USB connection to your PC . Select the ActiveSync option for synchronizing data , or use Disk Drive mode for transferring files . Wi-Fi Turn the Wi-Fi function on or off , scan for available wireless networks , see information about the active wireless network , and customize Wi-Fi settings . 13.3 Changing Basic Settings Display settings To manually adjust the screen brightness By default , your device automatically adjusts the screen brightness according to the ambient lighting . 1. Tap Start > Settings , and then tap All Settings . 2. On the System tab , tap Power > Backlight tab . 3. Clear the Auto adjust backlight check box . 4. Adjust the brightness by dragging the slider under On battery power and On external power . To automatically turn off the screen backlight when the device is not in use 1. Tap Start > Settings , and then tap All Settings . 2. On t
 he System tab , tap Power > Advanced tab . 3. In On battery power and On external power , specify the time before the screen backlight turns off . 236 Managing Your Device To increase or decrease the size of text on the screen 1. Tap Start > Settings , and then tap All Settings . 2. On the System tab , tap Screen > Text Size tab . 3. Move the slider to increase or decrease the text size . To re-calibrate the touch screen The device touch screen is calibrated before shipment . Re-calibrate the touch screen if it does not accurately respond to screen taps . 1. While the device is on , press and hold the VOLUME UP and VOLUME DOWN buttons , and then tap the screen to open the align screen . 2. Follow the on-screen instructions to complete the calibration . Device name Note The device name is used to identify your device when synchronizing with a computer , connecting to a network , or restoring information from a backup . If you synchronize multiple devices with the same computer , each
  device must have a unique name . To change the device name 1. Tap Start > Settings , and then tap All Settings . 2. On the System tab , tap About > Device ID tab . 3. Enter the device name . Notes \u2022 The device name must begin with a letter , consist of letters from A to Z , numbers from 0 to 9 , and cannot contain spaces . Use the underscore character to separate words . \u2022 If you enroll your device to your company\u2019s domain , the device name will be changed to be the same as the one used to enroll the device . Managing Your Device 237 Ringer settings To change the ring type and ring tone 1. Tap Start > Settings , and then tap Sound . 2. Scroll down and then tap Ring Type . Select the ring type you want to use , and then tap Done . 3. Tap Ringtone , and then select the ring tone you want to use . 4. Tap Done . Tips \u2022 You can use * . wav , * . mid , * . mp3 , * . wma , or other types of sound files that you downloaded from the Internet or copied from your computer . For a comp
 lete list of supported audio file formats , see \u201c Specifications \u201d in the Appendix . \u2022 To use your sound files as ring tones , copy them first to the /My Documents/My Ringtones folder on your device , and then select the sound from the Ring tone list of your device . For more information about copying files , see \u201c Copying and Managing Files \u201d in this chapter . To choose how to be notified about events or actions 1. Tap Start > Settings , and then tap Sound > Advanced . 2. On the Sounds tab , choose how you want to be notified by selecting the appropriate check boxes . 3. On the Notifications tab , in Event , tap a ring type and ring tone for that event or action from the lists below . Tip Turning off sound notifications helps conserve battery power . 238 Managing Your Device Phone services You can change the settings of various phone services for your device . Phone services may include Call Forwarding , Call Waiting , Voicemail , and more . Contact your wireless service 
 provider to find out about the availability of phone services for your device . To access and change settings of phone services 1. Tap Start > Settings , and then tap Communications . 2. On the Comm Manager screen , tap Phone > Phone services . 3. Select the service you want to use , and then tap Get Settings . 4. Choose your desired options , and then tap OK . 13.4 Using Connection Setup After you insert a new SIM card and power on your device , you can use Connection Setup to automatically configure your device\u2019s data connections , such as 3G/GPRS and WAP . Note For details on how to automatically configure the data connections , see \u201c Auto configuring data connection settings and importing SIM contacts \u201d in Chapter 1. To manually select the network operator to use for data connections You can manually select your network operator and allow Connection Setup to reconfigure your device\u2019s data connections settings based on the selected operator . 1. Tap Start > Settings , and
  then tap All Settings . 2. On the Connections tab , tap Connection Setup . 3. Select the network Country and Operator you want to use , then tap OK . Note If you choose a country and/or operator that is different from the installed SIM card network operator , you may not be able to access network services . Managing Your Device 239 4. When prompted to confirm whether you want to configure the data connection settings , tap Yes . 5. After Connection Setup completes the configuration , tap Restart . 13.5 Using Comm Manager Comm Manager lets you easily turn on and off the communication features of your device such as the phone , Bluetooth , Wi-Fi , and data connection . To access Comm Manager Tap Start > Settings , and then tap Communications . 1 2 3 4 5 6 7 8 9 10 11 12 240 Managing Your Device 1 Switch Airplane Mode on or off . See \u201c Turning the Phone Function On and Off \u201d in Chapter 2. Access the phone settings and change the PIN codes , configure phone services , and more . Co
 nnect a Bluetooth headset or other Bluetooth-enabled device . See Chapter 9 for details . Connect to a wireless network . See Chapter 8 for details . Open ActiveSync and adjust the push e-mail settings . See Chapter 7 for details . Open Connection Setup and automatically configure your data connection settings . 7 Switch the phone on or off . See \u201c Turning the Phone Function On and Off \u201d in Chapter 2. 8 Switch Bluetooth on or off . See Chapter 9 for details . 9 Switch Wi-Fi on or off . See Chapter 8 for details . 10 Switch between automatically receiving ( as items arrive ) or manually retrieving Outlook e-mail messages . See Chapter 7 for details . 11 Connect or disconnect active data connections ( 3G/GPRS , or EDGE ) . 12 Adjust the phone , Bluetooth , and Wi-Fi settings . 2 3 4 5 6 13.6 Using Task Manager Task Manager lets you view and stop running programs , configure the and enable the Quick Menu on the Home screen . button , To open Task Manager Tap the Quick Menu button o
 n the top-right corner of the Home screen to open the Quick Menu , and then tap . To configure the Exit ( ) button By default , running programs are shut down when you tap and hold the 1. On the Task Manager screen , tap the Button tab . Note When you clear Enable the \u201c X \u201d button to end running programs , tapping the Exit button only closes a program screen . The program is not ended and continues to run in the background . button . Managing Your Device 241 2. Choose the action for shutting down programs ( by tapping the by tapping and holding the button ) . 3. Tap OK . To add a running program to the exclusive list If you add a running program to the Exclusive Programs List , it will not be closed when you tap Stop Selected or Stop All and will not be listed in the Quick Menu on the Home Screen . 1. On the Task Manager screen , tap the Running tab . 2. Tap and hold the program name , and then tap Add Exclusive . Tip To remove a program from the exclusive list , tap and hold th
 e program on the Running tab , then tap Remove from Exclusive . button , or 13.7 Protecting Your Device Protecting your SIM card with a PIN You can protect your SIM card by requiring a PIN ( personal identification number ) every time the device is turned on . You will not be able to use the phone or the data connection until you enter the correct PIN . To enable the SIM card PIN 1. Tap Start > Settings , and then tap Communications . 2. On the Comm Manager screen , tap Phone > PIN Code . 3. Enter the PIN , tap OK , and then tap OK on the dialog box . Tips \u2022 To change the PIN , tap Change PIN on the Phone Settings screen . \u2022 Emergency calls can be placed at any time , without requiring a PIN . 242 Managing Your Device Protecting your device with a password You can help keep your data more secure by requiring a password every time the device is turned on . To set a device password 1. Tap Start > Settings , and then tap All Settings . 2. On the Personal tab , tap Lock . 3. Select 
 the Prompt if device unused for check box , then specify the idle time before the device requires a password . 4. In the Password type box , select the type of password you would like to use . Tip If your device is configured to connect to a network , use an alphanumeric password for increased security . 5. Enter the password in both the Password and Confirm boxes . 6. Tap OK . The next time the device is turned on , you will be prompted to enter your password . Notes \u2022 To ensure that you can always make emergency calls , do not begin your password with the digits of your local emergency number . \u2022 If you entered a hint , the hint will be displayed after the wrong password is entered five times . \u2022 Each time a wrong password is entered , the device response time gets longer until the device appears to be not responding . \u2022 If you forget your password , you can hard reset or clear the device memory . For more information about hard resetting the device and clearing the memory
  , see \u201c Resetting Your Device \u201d . Managing Your Device 243 Encrypting files on your storage card 2. On the System tab , tap Encryption . You can set your device to encrypt files as they are saved on the storage card . 1. Tap Start > Settings , and then tap All Settings . 3. Select Encrypt files when placed on a storage card . Notes \u2022 Encrypted files can only be read on the device that encrypted them . \u2022 Encrypted files are opened just like other files , provided you are using the device that encrypted them . There is no separate step for you to take to read these files . Important If Encrypt files when placed on a storage card is enabled , backup all files from the storage card before using Clear Storage , hard reset , or updating the operating system . Otherwise you will no longer be able to access the encrypted files on the storage card . Use ActiveSync or Windows Mobile Device Center to transfer files between your storage card and computer . After the procedure , copy yo
 ur files back to the storage card . 13.8 Resetting Your Device Performing a soft reset Perform a soft ( normal ) reset to clear all active program memory and shut down all active programs . This is useful when your device is running slower than normal , or a program is not performing properly . If a soft reset is performed when programs are running , unsaved work is lost . To perform a soft reset 1. Pull out the stylus . 2. Remove the back cover . For instructions , see \u201c To remove the back cover \u201d in Chapter 1. 244 Managing Your Device 3. Use the stylus tip to press the RESET button located at the upper-left side of the open back panel . Your device restarts and displays the Home screen . 4. Replace the back cover and insert the stylus back into its compartment . Performing a hard reset If a normal reset does not solve a system problem , you can perform a hard ( full ) reset . After a hard reset , the device is restored to its default settings\u2014the way it was when you first pu
 rchased it and turned it on . You will lose any programs you installed , data you entered , and settings you customized on your device , and only Windows Mobile� software and other pre-installed programs will remain . Warning ! our device will be set back to factory default settings . Make sure that any Y additional installed programs and/or user data have been backed up before you perform a hard reset . Important If Encrypt files when placed on a storage card is enabled , backup all files from the storage card before using Clear Storage , hard reset , or updating the operating system . Otherwise you will no longer be able to access the encrypted files on the storage card . Use ActiveSync or Windows Mobile Device Center to transfer files between your storage card and computer . After the procedure , copy your files back to the storage card . Managing Your Device 245 To perform a hard reset 1. With the device turned off , press and hold the TALK/SEND , END and POWER buttons until yo
 u see this message on the screen . This operation will delete all your personal data , and reset all settings to manufacturer default . Press VolUp to restore manufacturer default , or press other keys to cancel . 2. Release the TALK/SEND , END and POWER buttons . 3. Press the VOLUME UP button to perform the hard reset , or press any other button to cancel the reset . Using Clear Storage If you can access the device settings , you can use Clear Storage to reset your device back to factory default settings . 1. Tap Start > Settings , and then tap All Settings . 2. On the System tab , tap Clear Storage . 3. Enter \u201c 1234 \u201d and then tap Yes . 246 Managing Your Device 13.9 Windows Update Update Windows Mobile� with the latest security patches and fixes from Microsoft , if these are available . To set up Windows Update The first time you run Windows Update , you need to set up how you want Windows Update to check for updates . 1. Tap Start > Settings , and then tap All Settings . 2. 
 On the System tab , tap Windows Update . 3. On the Update Setup screen , tap Next . 4. Choose how you want to check for updates , then tap Next . Note If you select Automatic , you are asked if you want to use your current data plan to check for updates . Select or clear the Use my data plan to check for and download updates check box , and tap Next . 5. Tap Finish . To check for updates 1. Tap Start > Settings , and then tap All Settings . 2. On the System tab , tap Windows Update . 3. Tap Check Now . To change Windows Update options 1. Tap Start > Settings , and then tap All Settings . 2. On the System tab , tap Windows Update . 3. Tap Menu > Connections to change how the device connects to the Internet when checking for updates , or tap Menu > Change Schedule to change how Windows Update checks for Windows Mobile� updates . Managing Your Device 247 13.10 Connecting to an External Display Your device has a TV output which can be connected to an external display , such as a TV mon
 itor , that has an RCA cable socket . This allows you to output the device screen and audio to a big-screen display . Note You need an HTC TV Out cable to be able to connect your device to an external display . The HTC TV Out cable is sold separately . To connect the device to an external display 1. Turn off the external display . 2. Connect the ends ( red , white , and yellow RCA plugs ) of the HTC TV Out cable to the external display\u2019s RCA sockets . 3. Connect the mini-USB end of the HTC TV Out cable to the sync connector at the bottom of the device . The TV Out screen opens . 4. Choose the video standard to use when outputting the video and then tap Done . For example , if your video is in the NTSC standard and you select PAL-M , TV Out will automatically convert your video to PAL-M before it outputs to the external display . Note If you need to change the video standard option in the future , tap Start > Settings > All Settings > System tab > TV Out . 5. Tap the type of media 
 you want to output . 6. Turn on the external display and then switch it to the input signal that you connected the HTC TV Out cable to ( for example , Video 1 ) . The screen orientation is always in landscape mode while your device is connected to the external display . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/108TeleHTC_Manual_1_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/108TeleHTC_Manual_1_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/108TeleHTC_Manual_1_EN.txt.txt
new file mode 100644
index 0000000..0a79229
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/108TeleHTC_Manual_1_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Chapter 1 Getting Started 1.1 Getting to Know Your Touch Phone Inside the box \u2022 \u2022 \u2022 \u2022 \u2022 The product package includes : HTC Touch Pro2 Battery AC adapter Screen protector Wired stereo headset \u2022 \u2022 \u2022 \u2022 \u2022 USB sync cable Quick Start Guide Extra stylus Pouch Two accompanying discs Top panel POWER Press to turn off the display temporarily . To turn off the power , press and hold for about 3 seconds . For more information , see \u201c Starting Up \u201d in this chapter . 24 Getting Started Left panel Right panel VOLUME UP During a call or when playing music , press this button to increase the volume . Microphone VOLUME DOWN During a call or when playing music , press this button to lower the volume . Stylus Tap items , write , or draw on the touch screen . Getting Started 25 Front panel Notification LED* See description below . Earpiece Listen to a phone call from here . START Press to open or close the Start screen . See \u201c Start Screen \u201d in this chapter for details . TALK
 /SEND \u2022 Press to answer an incoming call or dial a number . \u2022 During a call , press and hold to turn the speakerphone on and off . \u2022 While viewing a text or e-mail message , press to call the sender of the message . Second Camera Use this camera for video call conversations or to take a self-portrait . Light Sensor Senses the ambient light and adjusts the screen backlight accordingly . BACK Press to return to the previous screen . END \u2022 Press to end a call or return to the Home tab . \u2022 Press and hold to lock the device . To customize the END key , see Buttons in Chapter 13. Zoom bar See \u201c Using the Zoom bar \u201d later in this chapter for details . * The Notification LED shows a : \u2022 Flashing green light when there are new SMS , MMS or e-mail messages , missed calls , or reminders . \u2022 Solid green light when the device is connected to the power adapter or a computer and the battery is fully charged . \u2022 Solid amber light when the battery is being charged . \u2022 Flashing 
 amber light when the remaining battery level reaches below 10 % . 26 Getting Started Hardware keyboard QWERTY Keyboard Slide open the touch screen panel to reveal the keyboard . See Chapter 4 for details . Touch Screen Getting Started 27 Back panel 3.2 Megapixel Camera See \u201c Taking Photos and Videos \u201d in Chapter 11 for details . Bottom panel Microphone Mute Press to mute or unmute the dual microphone . Dual Speaker Use this noise-cancelling dual speaker for conference calls . For more information about making conference calls , see \u201c Straight Talk\u2122 \u201d in Chapter 2. Back Cover Sync Connector/ Earphone Jack/TV Out \u2022 Connect the supplied USB cable to synchronize information . \u2022 Connect the power adapter to recharge the battery . \u2022 Connect the supplied USB stereo headset for hands-free conversation or for listening to audio media . \u2022 Connect an HTC composite video and audio cable ( purchased separately ) to output the device screen contents and device audio to an extern
 al viewing device . Strap Holder Attach a wrist strap , neck lanyard , or string accessory . 28 Getting Started 1.2 Installing the SIM Card , Battery , and Storage Card You need to remove the back cover before you can install the SIM card , battery and storage card . Also , make sure to always turn off the power before installing or replacing the SIM card and battery . To remove the back cover 1. Make sure your device is turned off . 2. Remove the stylus . Note You need to temporarily remove the stylus to be able to remove or replace the back cover . 3. Firmly hold the device with both hands and the front panel facing down . 4. Push the back cover up with your thumbs until it disengages from the device and then slide it up to remove . SIM card Note The SIM card contains your phone number , service details , and phonebook/ message memory . Your device supports both 1.8V and 3V SIM cards . Some legacy SIM cards may not function with your device . You should consult with your service p
 rovider for a replacement SIM card . There may be fees for this service . To install the SIM card 1. Remove the battery if it is installed . 2. Locate the SIM card slot , then insert the SIM card with its gold contacts facing down and its cut-off corner facing out the card slot . 3. Slide the SIM card completely into the slot . Cut-off corner Getting Started 29 To remove the SIM card 1. Remove the battery if it is installed . 2. Insert the tip of the stylus into the small hole above the SIM card slot , and then push the SIM card out with the stylus . 3. Slide the SIM card out from the slot with your other thumb or finger . Battery Your device comes with a rechargeable Lithium-ion battery and is designed to use only manufacturer-specified original batteries and accessories . Battery performance depends on many factors , including network configuration , signal strength , and usage patterns . For battery life estimates , see \u201c Specifications \u201d in the Appendix . Warning ! To reduce
  risk of fire or burns : \u2022 o not attempt to open , disassemble , or service the battery pack . D \u2022 o not crush , puncture , short external contacts , or dispose of in fire or water . D \u2022 Do not expose to temperatures above 60�C ( 140�F ) . \u2022 Replace only with the battery pack designated for this product . \u2022 ecycle or dispose of used battery as stipulated by local regulations . R 30 Getting Started To install the battery 1. Align the battery\u2019s exposed copper contacts with the battery connectors at the bottom part of the battery compartment . 2. Insert the contacts side of the battery first and then gently push the battery into place . 3. Replace the back cover and insert the stylus back into the stylus compartment . To remove the battery Lift out the battery from the groove at the upper right side of the battery compartment . Groove Storage card To have additional storage for your images , videos , music , and files , you can purchase a microSD\u2122 card and install it in
 to your device . To install a microSD card 1. Remove the stylus and then remove the back cover . 2. Locate the microSD card slot on the upper part of the exposed right panel . 3. Insert the microSD card into the slot with its gold contacts facing down until it clicks into place . Note To remove the microSD card , press it to eject it from the slot . Storage card slot Getting Started 31 1.3 Charging the Battery New batteries are shipped partially charged . Before you start using your device , it is recommended that you install and charge the battery . Some batteries perform best after several full charge/discharge cycles . Note Only the AC adapter and USB sync cable provided with your device must be used to charge the device . Connect the USB connector of the AC adapter to the sync connector on your device , and then plug in the AC adapter to an electrical outlet to start charging the battery . Charging is indicated by a solid amber light on the Notification LED . As the battery is b
 eing charged while the device is on , a charging icon also appears in the title bar of the Home screen . After the battery has been fully charged , a full battery icon appears in the title bar of the Home screen . Warning ! \u2022 o not remove the battery from the device while you are charging it using D the AC or car adapter . \u2022 s a safety precaution , the battery stops charging before it overheats . A 1.4 Starting Up After installing your SIM card and battery and charging the battery , you can now power on and start using your device . Turning your device on and off To turn on the device Press the POWER button . When you turn on your device for the first time , it installs customized settings and displays the SIM Card Changed screen . See \u201c Auto configuring data connection settings and importing SIM contacts \u201d in this chapter for details . 32 Getting Started To turn off the device 1. Press and hold the POWER button for a few seconds . 2. Tap Yes when prompted . Auto configuring
  data connection settings and importing SIM contacts After installing customized settings , the SIM Card Changed screen appears . Follow the on-screen instructions to automatically configure your device\u2019s data connections and/or import any contacts you have stored on your SIM card . The device restarts after all selected options are completed . Notes \u2022 The Import SIM contacts option does not appear if no contacts are found on your SIM card . \u2022 Touch Cancel if you prefer to configure your data connections or import your SIM contacts later . \u2022 If you choose to automatically configure your device\u2019s data connections and if your SIM card supports multiple network profiles , you will be prompted to select the profile to use . Switching to Sleep mode Briefly press the POWER button to turn off the display temporarily and switch your device to Sleep mode . Sleep mode suspends your device to a low power state while the display is off in order to save battery power . Your device auto
 matically goes into Sleep mode when you leave the device idle after a certain period of time . You are still able to receive messages and calls while your device is in Sleep mode . Your device wakes up automatically when you receive an incoming call . To wake up your device manually , briefly press the POWER button . Getting Started 33 1.5 Home Screen The main screen of the device is the TouchFLO\u2122 3D Home screen which allows finger-touch access to the most important functions such as contacts , text messages , e-mail , Internet , and more . For more information about using the TouchFLO 3D Home screen , see Chapter 3. Signal strength Connection status Notification Start screen Volume status Battery status Quick menu TouchFLO 3D Home screen Open the Phone screen ( See Chapter 2 for details ) Open the Camera . 34 Getting Started 1.6 Notifications When you touch or tap the icon area of the title bar , the Notifications screen opens to display your connection status and lists all your 
 pending notifications including missed calls , new SMS/MMS messages , and more . Touch an item on the Notifications screen to open the related screen or TouchFLO 3D tab . Open the related screen or tab . Dismiss all notifications . Close the Notifications screen . To close the Notifications screen , touch Close or press the BACK button . Getting Started 35 1.7 Status Icons The following are some of the status icons that you may see on the device . Icons that appear in this area Vibrate mode Sound on Sound off Icons that appear in this area Battery charging Battery full Battery very low Icons that appear in this area Connection Setup Multiple notifications ; tap the icon to view all New text messages ; SMS notification of voice mail Sending MMS Receiving MMS MMS sent Cannot send MMS New incoming MMS New Windows Live\u2122 message Missed call Icons that appear in this area Speakerphone on Roaming Alarm Other wireless network(s ) detected Wired headset connected Bluetooth on Bluetooth vis
 ible mode on Bluetooth stereo headset connected 36 Getting Started Icons that appear in this area GPRS available EDGE available HSPA available WCDMA/UMTS available Connected to a wireless network Connection is active Connection is not active Synchronization in progress Icons that appear in this area GPRS connecting GPRS in use EDGE connecting EDGE in use HSPA connecting HSPA in use WCDMA/UMTS connecting WCDMA/UMTS in use Maximum signal strength No signal Phone is turned off No phone service Searching for phone service Voice call in progress Call on hold Calls are forwarded Dialing while no SIM card is inserted No SIM card inserted Getting Started 37 1.8 Start Screen The Start screen , which can be opened by touching Start on the top-left corner of the screen or by pressing the START ( ) button , gives you one-touch access to all the programs and settings of your Windows Mobile-powered device . You can also add the shortcut icons of your favorite programs in the Start screen so you c
 an open them quickly . Open the Start screen . Open the TouchFLO 3D People tab . Open the TouchFLO 3D Calendar tab . Open ActiveSync . Open the TouchFLO 3D Home tab . See \u201c Home \u201d in Chapter 3 for details . Open the TouchFLO 3D Settings tab and adjust the device settings . See \u201c Settings \u201d in Chapter3 for details . Swipe upward the screen to view or add more program icons . Open the corresponding program . Add a program icon . Access all programs on your device . Note Remove a program icon . The Home , People , Calendar , and Settings icons are permanent icons and cannot be removed or replaced with other program icons . 38 Getting Started To add a program icon to the Start screen 1. Touch an empty slot to open the Add Program screen that lists all programs on your device . 2. Touch the program you want to add to the Start screen . 3. When you have filled up the current Start screen with programs , scroll down the screen to access more empty slots and add more program icons .
  To scroll down , press and hold on the screen and then drag your finger upward . To scroll back up , press and hold on the screen and then drag your finger downward . To remove or replace a program icon on the Start screen 1. On the Start screen , touch Remove . 2. Touch the program icon you want to remove or replace to clear the slot . 3. Touch Done . 4. If you are replacing the slot with another program icon , follow the steps in \u201c To add a program icon to the Start screen \u201d . To launch a program Do any of the following on the Start screen : \u2022 \u2022 Touch the program icon ; or Touch All Programs , and then touch the program in the list . Tip The programs list is long . If you do not see the desired program on the current screen , drag your finger upward to view more programs . Getting Started 39 1.9 Quick Menu The Quick menu , which can be opened from the top right corner of the screen , shows the currently running programs . You can quickly switch between running programs , 
 stop programs , and see how much program memory is being used . Customize Task Manager settings . ( See \u201c Using Task Manager \u201d in Chapter 13. ) Touch this icon to open or close the Quick Menu . Stop all currently running programs . Shows the total percentage of program memory in use . Touch to open the Memory settings screen . \u2022 To switch to a running program , touch the program name . \u2022 To stop a running program , touch . 40 Getting Started 1.10 Adjusting the Volume A single volume control adjusts both the system volume ( notifications and audio/ video playback volume ) and the phone ring volume of your device . To set a single volume 1. Press the VOLUME UP/DOWN button on the left panel of your device to open the Volume screen . 2. To increase or decrease the volume level , drag your finger up or down the volume bars , or press the VOLUME UP/DOWN button . You can also : \u2022 Touch Silent to mute the volume . \u2022 Touch Vibrate to switch the device to Vibrate mode . 3. The Vol
 ume screen automatically closes . You can also tap an empty area of the Volume screen to close it . Note Use the VOLUME UP/DOWN button on the left panel of your device to adjust the conversation phone volume during an ongoing call . Volume bars To adjust the system and ring volumes separately 1. On the Home screen , slide to the Settings tab , and then touch Sound . 2. Turn off the Single Volume option . 3. Press the VOLUME UP/DOWN button . 4. To switch between the Ring Volume and System Volume screens , touch Ring or System at the top of the screen . 5. To adjust the volume level , follow steps 2 to 3 in \u201c To set a single volume \u201d . Getting Started 41 1.11 Using Finger Gestures You can use finger gestures to scroll and pan on the touch screen . Finger scrolling Finger scrolling can be used to scroll up and down web pages , documents , and lists such as the contacts list , file list , message list , calendar appointments list , and more . When finger scrolling , swipe or flick y
 our finger on the touch screen . To scroll up and down \u2022 To scroll down , swipe your finger upward on the touch screen . To scroll up , swipe your finger downward on the touch screen . \u2022 To auto-scroll , flick your finger upward or downward on the touch screen . Touch the screen to stop scrolling . 42 Getting Started To scroll left and right \u2022 To scroll towards the right , swipe your finger to the left . To scroll towards the left , swipe your finger to the right . \u2022 To auto-scroll , flick your finger left or right . Touch the screen to stop scrolling . Finger panning You can use finger panning to move to and view other parts of a zoomed photo , web page , document , or an e-mail . To pan , touch and hold on the touch screen and then drag your finger in any direction . Getting Started 43 1.12 Using the Zoom Bar The Zoom bar located below the touch screen is a handy zoom control that is unique on your device . When you are viewing a web page in Opera Mobile or a photo in Albu
 m , you can zoom in by sliding your finger from left to right across the Zoom bar , or zoom out by sliding from right to left . What\u2019s more , you can also instantly enlarge or shrink the text size when you open and read messages or Word documents . Slide your finger from left to right across the Zoom bar to increase the text size , or slide from right to left to decrease . To increase or decrease the text size more , slide across the Zoom bar again . Zoom bar 1.13 Connecting Your Device to a Computer Whenever you connect your device to the computer with the USB cable , the Connect to PC screen displays and prompts you to choose the type of USB connection : ActiveSync Select this mode when you need to do the following : \u2022 \u2022 Synchronize Outlook e-mail messages , contacts , calendar appointments , files , and other data between your device and your computer . Have read and write access to the whole device . For more information about using ActiveSync , see Chapter 5. 44 Getting S
 tarted Disk Drive This mode is available for selection only when you installed a microSD card on your device . Select this mode when you want to use your storage card as a USB thumb drive , and copy files between your device\u2019s storage card and your computer faster . When Disk Drive mode is enabled while your device is connected to the computer : \u2022 \u2022 You will not be able to use your device to access files from the storage card or run applications that are installed on the storage card . Your computer will only connect to the device\u2019s storage card . When you access your device from the computer , you will only see the contents of the device\u2019s storage card . Internet Sharing When you do not have Internet access on your computer , you can select this mode to use your device as a modem for your computer . For more information about using Internet Sharing , see \u201c Using Your Device as a Modem ( Internet Sharing ) \u201d in Chapter 8. 
\ No newline at end of file


[19/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Teli/117TeliApple_iPhone4_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Teli/117TeliApple_iPhone4_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Teli/117TeliApple_iPhone4_EN.txt.txt
new file mode 100644
index 0000000..7756494
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Teli/117TeliApple_iPhone4_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeffiPhone 4 Important Product Information Guide This Important Product Information Guide contains safety and handling , regulatory , software license , and warranty information for iPhone . Look for recycling , disposal , and other environmental information in the iPhone User Guide at : support . apple . com/manuals/iphone To avoid injury , read all operating instructions and the following safety information before using iPhone . For detailed operating instructions , read the iPhone User Guide on your iPhone by visiting help . apple . com/iphone or using the iPhone User Guide bookmark in Safari . For downloadable versions of the latest iPhone User Guide and this Important Product Information Guide , visit : support . apple . com/manuals/iphone Important Safety and Handling Information WARNING : Failure to follow these safety instructions could result in fire , electric shock , or other injury or damage to iPhone or other property . Carrying and Handling iPhone iPhone contains sensit
 ive components . Do not drop , disassemble , open , crush , bend , deform , puncture , shred , microwave , incinerate , paint , or insert foreign objects into iPhone . Do not use iPhone if it has been damaged\u2014for example , if iPhone is cracked , punctured , or damaged by water . The front and back covers of iPhone are made of glass . This glass could break if iPhone is dropped on a hard surface or receives a substantial impact or is crushed , bent , or deformed . If the glass chips or cracks , do not touch or attempt to remove the broken glass . Stop using iPhone until the glass is replaced by Apple or an Apple Authorized Service Provider . Glass cracked due to misuse or abuse is not covered under the warranty . If you are concerned about scratching or abrasion , you can use one of the many cases sold separately . Keeping the Outside of iPhone Clean Clean iPhone immediately if it comes into contact with any contaminants that may cause stains , such as ink , dyes , makeup , dirt , 
 food , oils , and lotions . To clean iPhone , unplug all cables and turn off iPhone ( press and hold the Sleep/Wake button , and then slide the onscreen slider ) . Then use a soft , slightly damp , lint-free cloth . Avoid getting moisture in openings . Don\u2019t use window cleaners , household cleaners , aerosol sprays , solvents , alcohol , ammonia , or abrasives to clean iPhone . The front and back glass surfaces have an oleophobic coating . To remove fingerprints , simply wipe these surfaces with a soft , lint-free cloth . The ability of this coating to repel oil will diminish over time with normal usage , and rubbing the screen with an abrasive material will further diminish its effect and may scratch the glass . Avoiding Water and Wet Locations Do not use iPhone in rain , or near washbasins or other wet locations . Take care not to spill any food or liquid on iPhone . In case iPhone gets wet , unplug all cables , turn off iPhone ( press and hold the Sleep/Wake button , and then s
 lide the onscreen slider ) before cleaning , and allow it to dry thoroughly before turning it on again . Do not attempt to dry iPhone with an external heat source , such as a microwave oven or hair dryer . Damage to iPhone caused by contact with liquid is not covered under the warranty . Repairing or Modifying iPhone Never attempt to repair or modify iPhone yourself . iPhone does not contain any user-serviceable parts , except for the SIM card and SIM tray . Disassembling iPhone , including the removal of external screws and back cover , may cause damage that is not covered under the warranty . Service should only be provided by Apple or an Apple Authorized Service Provider . If you have questions , contact Apple or an Apple Authorized Service Provider . For service information , go to : www . apple . com/support/iphone/service/faq Battery Replacement Do not attempt to replace the rechargeable battery in iPhone yourself . The battery may only be replaced by Apple or an Apple Authori
 zed Service Provider . For more information about battery replacement service , go to : www . apple . com/support/iphone/service/battery Charging iPhone To charge iPhone , only use the Apple Dock Connector to USB Cable with an Apple USB Power Adapter or a high-power USB port on another device that is compliant with the USB 2.0 or 1.1 standard , another Apple-branded product or accessory designed to work with iPhone , or a third-party accessory certified to use the Apple \u201c Works with iPhone \u201d or \u201c Made for iPhone \u201d logo . Read all safety instructions for any products and accessories before using with iPhone . Apple is not responsible for the operation of , or any damage caused by , third-party accessories or their compliance with safety and regulatory standards . When you use the Apple USB Power Adapter to charge iPhone , make sure that the power adapter is fully assembled before you plug it into a power outlet . Then insert the Apple USB Power Adapter firmly into the power o
 utlet . Do not connect or disconnect the Apple USB Power Adapter with wet hands . The Apple USB Power Adapter may become warm during normal use . Always allow adequate ventilation around the Apple USB Power Adapter and use care when handling . Unplug the Apple USB Power Adapter if any of the following conditions exist : The power cord or plug has become frayed or damaged . The adapter is exposed to rain , liquid , or excessive moisture . The adapter case has become damaged . You suspect the adapter needs service or repair . You want to clean the adapter . Avoiding Hearing Damage Permanent hearing loss may occur if the receiver , earbuds , headphones , speakerphone , or earpieces are used at high volume . Set the volume to a safe level . You can adapt over time to a higher volume of sound that may sound normal but can be damaging to your hearing . If you experience ringing in your ears or muffled speech , stop listening and have your hearing checked . The louder the volume , the less
  time is required before your hearing could be affected . Hearing experts suggest that to protect your hearing : Limit the amount of time you use the receiver , earbuds , headphones , speakerphone , or earpieces at high volume . Avoid turning up the volume to block out noisy surroundings . Turn the volume down if you can\u2019t hear people speaking near you . For information about how to set a maximum volume limit on iPhone , see the iPhone User Guide . Emergency Calls You should not rely on wireless devices for essential communications , such as medical emergencies . Use of iPhone to call emergency services may not work in all locations or all operating conditions . Emergency numbers and services vary by region , and sometimes an emergency call cannot be placed due to network availability or environmental interference . Some cellular networks may not accept an emergency call from iPhone if it does not have a SIM , if the SIM is PIN-locked , or if you have not activated your iPhone . D
 riving and Riding Safely Use of iPhone alone or with headphones ( even if used only in one ear ) while driving a vehicle or riding a bicycle is not recommended and is illegal in some areas . Check and obey the laws and regulations on the use of mobile devices like iPhone in the areas where you drive or ride . Be careful and attentive while driving or riding a bicycle . If you decide to use iPhone while driving , keep in mind the following guidelines : 1 Give full attention to driving or riding and to the road . Using a mobile device while driving or riding may be distracting . If you find it disruptive or distracting while operating any type of vehicle , riding a bicycle , or performing any activity that requires your full attention , pull off the road and park before making or answering a call . Get to know iPhone and its features such as Voice Control , Favorites , Recents , and Speaker . These features help you to place your call without taking your attention off the road . See t
 he iPhone User Guide for more information . Use a hands-free device . Take advantage of one of the many compatible hands-free accessories available for use with iPhone . Use of a hands-free device may be required in some areas . Position iPhone within easy reach . Keep your eyes on the road . If you get an incoming call at an inconvenient time , let your voicemail answer it for you . Try to plan calls when your car will be stationary . Assess the traffic before dialing , or place calls when you are not moving or before pulling into traffic . Let the person you are speaking with know you are driving . If necessary , suspend the call in heavy traffic or hazardous weather conditions . Driving in rain , sleet , snow , ice , fog , and even heavy traffic can be hazardous . Do not text , email , take notes , look up phone numbers , or perform any other activities that require your attention while driving . Composing or reading texts and emails , jotting down a to-do list , or flipping thro
 ugh your address book takes attention away from your primary responsibility , driving safely . Do not engage in stressful or emotional conversations that may be distracting . Make people you are talking with aware you are driving and suspend conversations that have the potential to divert your attention from the road . Navigating Safely Do not rely on iPhone applications that provide maps , digital compass headings , orientation information , directions , or location-based navigation to determine precise locations , proximity , orientation , distance , or direction . These applications should only be used for basic navigation assistance . Maps , directions , and location-based applications depend on data services . These data services are subject to change and may not be available in all geographic areas , resulting in maps , digital compass headings , directions or location-based information that may be unavailable , inaccurate , or incomplete . iPhone contains an internal digital 
 compass located in the upper-right corner of iPhone . The accuracy of digital compass headings may be negatively affected by magnetic or other environmental interference , including interference caused by the close proximity of the magnets contained in the iPhone earbuds . Never rely solely on the digital compass for determining direction . Compare the information provided on iPhone to your surroundings and defer to posted signs to resolve any discrepancies . Do not use location-based applications while performing activities that require your full attention . For important information about driving safety , see the \u201c Driving and Riding Safely \u201d section . Always comply with posted signs and the laws and regulations in the areas where you are using iPhone . For Vehicles Equipped with an Air Bag An air bag inflates with great force . Do not store iPhone or any of its accessories in the area over the air bag or in the air bag deployment area . Seizures , Blackouts , and Eyestrain A 
 small percentage of people may be susceptible to blackouts or seizures ( even if they have never had one before ) when exposed to flashing lights or light patterns such as when playing games or watching video . If you have experienced seizures or blackouts or have a family history of such occurrences , you should consult a physician before playing games ( if available ) or watching videos on your iPhone . Discontinue use of iPhone and consult a physician if you experience headaches , blackouts , seizures , convulsion , eye or muscle twitching , loss of awareness , involuntary movement , or disorientation . To reduce risk of headaches , blackouts , seizures , and eyestrain , avoid prolonged use , hold iPhone some distance from your eyes , use iPhone in a well-lit room , and take frequent breaks . Choking Hazards iPhone contains small parts , which may present a choking hazard to small children . Keep iPhone and its accessories away from small children . Repetitive Motion When you per
 form repetitive activities such as typing or playing games on iPhone , you may experience occasional discomfort in your hands , arms , shoulders , neck , or other parts of your body . Take frequent breaks and if you have discomfort during or after such use , stop use and see a physician . Potentially Explosive Atmospheres Turn off iPhone ( press and hold the Sleep/Wake button , and then slide the onscreen slider ) when in any area with a potentially explosive atmosphere . Do not charge iPhone , and obey all signs and instructions . Sparks in such areas could cause an explosion or fire , resulting in serious injury or even death . Areas with a potentially explosive atmosphere are often , but not always , marked clearly . Potential areas may include : fueling areas ( such as gas stations ) ; below deck on boats ; fuel or chemical transfer or storage facilities ; vehicles using liquefied petroleum gas ( such as propane or butane ) ; areas where the air contains chemicals or particles (
  such as grain , dust , or metal powders ) ; and any other area where you would normally be advised to turn off your vehicle engine . Using Connectors and Ports Never force a connector into a port . Check for obstructions on the port . If the connector and port don\u2019t join with reasonable ease , they probably don\u2019t match . Make sure that the connector matches the port and that you have positioned the connector correctly in relation to the port . Accessories and Wireless Performance Not all iPod accessories are fully compatible with iPhone . Turning on Airplane Mode on iPhone may eliminate audio interference between iPhone and an accessory . While Airplane mode is on , you cannot make or receive calls or use features that require wireless communication . Under some conditions , certain accessories may affect iPhone wireless performance . Reorienting or relocating iPhone and the connected accessory may improve wireless performance . Keeping iPhone Within Acceptable Temperatures iPh
 one is designed to be operated in temperatures between 0� and 35� C ( 32� to 95� F ) and stored in temperatures between -20� and 45� C ( -4� to 113� F ) . Low- or high-temperature conditions might temporarily shorten battery life or cause iPhone to temporarily stop working properly . Leaving iPhone in a parked vehicle or in direct sunlight can cause iPhone to exceed these storage or operating temperature ranges . Avoid dramatic changes in temperature or humidity when using iPhone as condensation may form on or within iPhone . When you\u2019re using iPhone or charging the battery , it is normal for iPhone to get warm . The exterior of iPhone functions as a cooling surface that transfers heat from inside the unit to the cooler air outside . Exposure to Radio Frequency Energy iPhone contains radio transmitters and receivers . When on , iPhone receives and sends out radio frequency ( RF ) energy through its antennas . The iPhone cellular antenna is located at the bottom edge of iPh
 one , to the left of the Home button . The Wi-Fi and Bluetooth� antenna is located at the top edge of iPhone , to the right of the headset jack . For optimal mobile device performance and to be sure that human exposure to RF energy does not exceed the FCC , IC , and European Union guidelines , always follow these instructions and precautions : When on a call using the built-in audio receiver in iPhone , hold iPhone with the dock connector pointed down toward your shoulder to increase separation from the antenna . When using iPhone near your body for voice calls or for wireless data transmission over a cellular network , keep iPhone at least 15 mm ( 5/8 inch ) away from the body , and only use carrying cases , belt clips , or holders that do not have metal parts and that maintain at least 15 mm ( 5/8 inch ) separation between iPhone and the body . iPhone is designed and manufactured to comply with the limits for exposure to RF energy set by the Federal Communications Commission ( FC
 C ) of the United States , Industry Canada ( IC ) of Canada , and regulating entities of Japan , the European Union , and other countries . The exposure standard employs a unit of measurement known as the specific absorption rate , or SAR . The SAR limit applicable to iPhone set by the FCC is 1.6 watts per kilogram ( W/kg ) , 1.6 W/kg by Industry Canada , and 2.0 W/kg by the Council of the European Union . Tests for SAR are conducted using standard operating positions ( i. e. , at the ear and worn on the body ) specified by these agencies , with iPhone transmitting at its highest certified power level in all tested frequency bands . Although SAR is determined at the highest certified power level in each frequency band , the actual SAR level of iPhone while in operation can be well below the maximum value because iPhone adjusts its cellular transmitting power based in part on proximity to the wireless network . In general , the closer you are to a cellular base station , the lower th
 e cellular transmitting power level . iPhone has been tested , and meets the FCC , IC , and European Union RF exposure guidelines for cellular , Wi-Fi , and Bluetooth operation . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/118TelsGoog_Answer_0545_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/118TelsGoog_Answer_0545_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/118TelsGoog_Answer_0545_EN.txt.txt
new file mode 100644
index 0000000..26fdd61
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/118TelsGoog_Answer_0545_EN.txt.txt
@@ -0,0 +1,2 @@
+
+As you type in the search box on Google Web Search , Google 's autocomplete algorithm offers searches that might be similar to the one you 're typing . Start to type [ new york ] -- even just [ new y ] -- and you 'll be able to pick searches for New York City , New York Times , and New York University ( to name just a few ) . Why it 's helpful Rest your fingers . The algorithm predicts search queries in real-time , so typing [ golden gate b ] and then clicking 'golden gate bridge ' is faster and easier than typing it out . Catch a mistake . Did you mean : Melbourne Australia ? Start searching for [ melborn ] and Google 's algorithm will present more common spellings for what you might be trying to find . Repeat a favorite search . If you 're signed in to your Google Account and have Web History enabled , the algorithm may show some predicted queries based on searches that you 've done in the past . Data that you send to Google is protected by Google 's privacy policy . Find other us
 eful information . Even if you 've disabled Google Instant , there are some features in autocomplete to connect you immediately with the information you 're looking for . For example , if you type [ weather brus ] you may see the current weather in Brussels , Belgium , described right in the list of search terms . This also works for flight status , local time zones , area codes , package tracking , certain answers , definitions , calculations , plus currency and unit conversions . How autocomplete works As you type , Google 's algorithm predicts and displays search queries based on other users ' search activities . In addition , if you 're signed in to your Google Account and have Web History enabled , you may see search queries from relevant searches that you 've done in the past . All of the predicted queries that are shown in the drop-down list have been typed previously by Google users . For certain queries , Google will show separate predictions for just the last few words . B
 elow the word that you 're typing in the search box , you 'll see a smaller drop-down list containing predictions based only on the last words of your query . While each prediction shown in the drop-down list has been typed before by Google users , the combination of your primary text along with the completion may be unique . Predicted queries are algorithmically determined based on a number of purely objective factors ( including popularity of search terms ) without human intervention . The autocomplete data is updated frequently to offer fresh and rising search queries . Does Google exclude any users ' searches from autocomplete ? The search queries that you see as part of autocomplete are a reflection of the search activity of all web users . Just like the web , the search queries presented may include silly or strange or surprising terms and phrases . While we always strive to neutrally and objectively reflect the diversity of content on the web ( some good , some objectionable 
 ) , we also apply a narrow set of removal policies for pornography , violence , hate speech , and terms that are frequently used to find content that infringes copyrights . Why do some of the searches seem so strange ? In some cases , there may be a search term that seems surprising to you , but after doing some searching on the web , you may discover that it 's a popular phrase online for some reason that you did n't anticipate . Queries in autocomplete are algorithmically determined based on a number of objective factors ( including search term popularity ) without manual intervention . Why do n't I see any predictions for a particular topic ? If no predicted searches appear for a particular word or topic , it 's likely due to one of the following reasons : The search term is not popular enough . Queries that are n't popular are less likely to be useful in autocomplete . A minimum algorithmic threshold for popularity helps us fight spam and improves the likelihood that the search 
 queries you do see are relevant to your search . We also make regionally appropriate adjustments to autocomplete , so the predicted queries that you see on Google Australia ( google . com . au ) or Google France ( google . fr ) may be different from those you see on Google . com . The search term is too fresh . It can take some time for newly popular search terms to consistently appear in autocomplete . While you 'll usually be able to discover new queries within an hour , it can sometimes take a few days to a few weeks for recently popular search queries to start appearing in autocomplete . The search term violates our autocomplete policies . We aim to provide you with relevant search queries , but we exclude a narrow class of search queries related to pornography , violence , hate speech , and copyright infringement . The search term was mistaken for a policy violation . Our policy review process is neither perfect nor instantaneous . There might be occasions where the system aims
  to exclude a query in one language that would be perfectly fine in another language . For example , we might inadvertently exclude a compound word because it includes a translation of a bad word from another language . Control personalized predictions If you do n't want to see search queries based on your web history , you can control how personalized autocomplete works . Here are your options : Sign out of your Google Account while you search on Google . If you are n't signed in to a Google Account , your search experience may be customized based on past search information linked to a cookie on your browser . Learn how to turn off these customizations . Remove particular searches from your Web History at www . google . com/history or by clicking the " Remove " links that appear beside personalized predictions . Remove Web History from your Google Account . Ask a question or report a problem with autocomplete If you see a problem with the predictions in autocomplete , it 's probabl
 y caused by one of the scenarios described above and we are already working to address it . If you have feedback or questions about autocomplete , you can post them in the Google Web Search Help Forum . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/119TelsGoog_Answer_0a46_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/119TelsGoog_Answer_0a46_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/119TelsGoog_Answer_0a46_EN.txt.txt
new file mode 100644
index 0000000..e12eea7
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/119TelsGoog_Answer_0a46_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Use Google Realtime Search to find relevant real-time updates from online services including Twitter , FriendFeed , Facebook , MySpace , indenti . ca , TwitArmy , and Jaiku . Search for only this kind of content by visiting google . com/realtime . You can also search for real-time updates by clicking the Realtime link ( sometimes found under the " More " menu ) on the side of any search results page . In this Realtime view , you 'll see recent posts that are relevant to your search . Tweets and other short-form updates create an ongoing commentary that can provide valuable insights into what 's happened and how people have reacted . Here are a few cool features and options within the Realtime view : Top Links The Top Links section features the most cited links for your search , helping you find the most relevant content that was shared through these real-time updates . You might find content like a new video that 's gone viral or the latest news story that was commonly shared . Conv
 ersations Use the Full conversation option to see the back and forth dialog of a Twitter conversation . The Full conversation snippet will automatically appear next to tweets that have @replies associated with them . Tweets are organized from oldest to newest and are indented so you can quickly see how the conversation developed . The Full conversation option is currently available only in certain languages . Replay Use Google to search and explore the archive of past Twitter content . By replaying tweets , you can explore any topic that people have discussed on Twitter . For example , see what people were saying about a certain news event , how opinions of a celebrity changed over time , or what event made your favorite bookstore so crowded last weekend . Once you 're in the Realtime view , make sure that the Latest option is selected in the side panel ( it will be selected by default ) . You 'll see a chart on the right side of the page that shows how the relative volume of tweets
  about your search have changed over time . Click any point in the chart to see a replay of what people were saying publicly on Twitter about a given topic at that given time . Note that we currently show this historical view only for Twitter content . Filter your real-time results -- Sort by date : When you search for real-time updates , you 'll see the Latest view by default . This option shows content ordered by date and time , so you 'll see the most recent updates listed first . If you want to see results ranked by their relevance to your search terms and not by date , click the Any time link in the left panel . Results in this view are ranked based on algorithms to determine the most relevant updates regardless of when they were written . -- Sort by images : If you want to see only results that include image thumbnails , click the Updates with images link in the left panel . -- Sort by location : Use geographic location to help find the most relevant real-time results . When a
 vailable , the location is shown next to the timestamp for each update . This location can be the location set by the user in their profile or a location extracted from explicit latitude and longitude attached to an update . In the Realtime view , you can use the filtering option in the side panel to see results from Anywhere , Nearby , or a Custom location that you define . If you choose Nearby , you 'll see content that 's customized to your location . The geographic location filter is currently available only in certain languages . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/120TelsGoog_Answer_0bcf_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/120TelsGoog_Answer_0bcf_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/120TelsGoog_Answer_0bcf_EN.txt.txt
new file mode 100644
index 0000000..eb41cf8
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/120TelsGoog_Answer_0bcf_EN.txt.txt
@@ -0,0 +1,2 @@
+
+As you start to type your search terms , Google Instant automatically shows results for a popular search that begins with those letters . If you do n't see the results you want , just keep typing and the results will dynamically update . Connect to the information you need faster , even before you finish typing your search ! How it works The basics of Google search are the same , including how results are ranked and how Google determines relevant results . What has changed is the path you can take to get to relevant results . Here 's what happens when you search : Start to type . As you type a search on Google , the homepage automatically starts displaying your search results -- no need to press Enter . A list of predicted queries is generated using the autocomplete algorithm . These predictions are displayed in a drop-down menu below the search box . See results . The algorithm tries to predict what the rest of your query might be based on popular queries typed by other users . The
  first prediction is shown in light grey in the search box , and the search results you see instantly will be for that predicted search . If instead you want to see results for only the text you 've typed , just press Enter or click the Search button . Type more , see more . If you continue to type , Google dynamically updates the search results to match the first prediction for what you 've typed . Refine your search . Do n't find the information or website you want ? It 's easy to refine your search after seeing what results appear . Here are some options : If you see what you 're looking for in the list of predicted queries , click the query to see relevant results . Use your keyboard 's down arrow key to scroll through the list of predicted queries , and see results appear for each query you pause over . Continue typing your query until the results show what you 're looking for . As always , you 're still able to type your full query and press Enter or click the Search button . 
 Why do n't I see results appear as I type ? You wo n't see results appear as you type if any of the following is true : you search from a page that 's not the Google homepage or a search results page you search from your iGoogle page rather than the classic homepage you search using Google SSL you use an Internet browser that 's not one of these supported browsers : Firefox 3 and later versions , Safari 5 for Mac and later versions , Internet Explorer 8 , and Google Chrome 5 and later versions you previously disabled autocomplete you previously opted out of Google Instant you use a Google domain or interface language that does not yet support Google Instant In addition , users on particularly slow Internet connections wo n't see dynamic results since we want to make sure that your search experience is as fast as possible . Learn more about this and similar issues . Lastly , certain types of search queries that may be inappropriate will not trigger results to appear automatically . I
 n these situations , you 'll see a message asking you to click Enter to perform the search . The benefits Get all the benefits of autocomplete and more : Save time -- Get relevant search results while you type so you spend less time searching and can go right to the web content you want . Type less -- Dynamically generated results let you stop typing as soon as you see what you need . We estimate that this will reduce the average number of keystrokes by 9 per search . Search smarter -- Like a conversation with the search engine , get instant feedback on whether your search terms return the type of results you want . If not , just add another word or two to see if the results are closer to what you 're looking for . Learn how to get the most out of Google Instant Can I opt out ? If you do n't want to see results as you type , you can opt out by accessing your search settings under the gear icon on any search results page . When you opt out , we 'll save that preference on a cookie , 
 so you 'll stayed opted out until the cookie is cleared . Note that the preference will only apply to that particular computer and Internet browser , and it is not tied to your Google Account . If you 're concerned about seeing inappropriate content , you can enable SafeSearch at the " strict filtering " level to help avoid sites that contain pornography , explicit sexual content , profanity , and types of hate content . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/121TelsGoog_Answer_1851_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/121TelsGoog_Answer_1851_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/121TelsGoog_Answer_1851_EN.txt.txt
new file mode 100644
index 0000000..8abec33
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/121TelsGoog_Answer_1851_EN.txt.txt
@@ -0,0 +1,2 @@
+
+If you want to filter and customize the search results you see , use the options along the side of the results page . For example , you can choose to just see blogs updated within the last 24 hours or photos of a certain size . How to use search options and tools On the side of your search results page , click a filtering option to customize the search results page you 're looking at . We 'll automatically show you the tools and filtering options that are most relevant to your search . To see more , click More and More search tools to see all the available filters and views you can apply to that search . Since your options will vary based on the search and the filters you 've already used , you wo n't see all the options all the time . Try clicking on two or more options to form a more complex , more focused filter . To go back to standard unfiltered Google results , click Reset tools at the bottom of the panel . Filter results by type of content The top group of options will filter
  results by the type of web content you 'll see . Everything : By default , your Google search will show unfiltered results that can include all the types of content below . Images : See only results from Google Images . News : See only results from Google News . Books : See only results from Google Books including reviews , excerpts , and where you can buy the book . Videos : See only video results using Google 's Video Search . Blogs : See only results from Google Blog Search . Places : See only results from Google Place Pages . Shopping : See only results from Google Product Search . ( With this option , you 'll see results within the Product Search site rather than in a Google search results page . ) Realtime : See only real-time updates from micro-blogs like Twitter , FriendFeed , Jaiku and others . Learn more about Google real-time search Discussions : See what people are saying in discussion groups , forums , and question-and-answer sites . Recipes : See only results for reci
 pes . Customize and filter these results to show recipes with your ideal ingredients , cook time and calorie count . This feature is currently not available in all languages . Learn more about Recipe View You wo n't always see all of these options , so click More below your list to see all the available types for that particular search . Customize your results by location The location used to customize your results is clearly indicated on the left side of your results page . Our location detection technology will automatically set a location for you , but you can easily specify an alternate location . Filter results by other criteria Based on your search and the Google property you 've selected , you 'll see a customized list of filtering tools . We 'll automatically show you the tools that seem most relevant to your search , so you wo n't always see all of these tools . Here are a few tools that you might see : Publish date Limit results according to when they were published on the
  Web . Google tries to estimate the publication date for a page by using information such as the date when Google first indexed the page . See the options Any time : By default , your search results will not be filtered by date . Latest : Sometimes the top results that show up are a little dated , although they 're still relevant . The Latest option shows you results that are both relevant and fresh . Google automatically optimizes the time period used for your filter to make sure that the most relevant and recent results are included . ( This option is only available when Everything is selected in the first group of options . ) Past 24 hours / Past week / Past month / Past year : Use these pre-defined ranges of times to find webpages . For instance , the Past 24 hours option will help you find information about current events . Custom range : This option lets you define your own date range . Enter the dates you want to use in the format mm/dd/yyyy , then click Search . This option 
 works best for pages published after 2001 ; to find historical mentions of dates , try using the Timeline tool . Personalization Limit the results to content that is more personalized to you . For example , see content from your friends ( Social ) or content from a geographical area ( Nearby ) . See the options Social : If you 're signed in to Google , see only social search results . You 'll see publicly-accessible content from people in your social circle that is relevant to your search . The idea is that content from your friends and social contacts is often more relevant to you than content from strangers . Nearby : Only see results that might be especially relevant for your preferred location indicated on the side of your results page . You can easily specify your preferred location . Web History : Limit the results to pages you have or have not already visited . If you 're signed in to your Google Account and have Web History enabled , you can use this tool to restrict your re
 sults to pages you either have , or have n't , visited in the past while Web History was enabled . Want to find something new ? Try Not yet visited . Need to remember a helpful site from last month ? Visited pages will focus your search . Display options This section lets you change the way results are displayed . The default Standard view shows you regular Google results , where each result is composed of the page title and a few lines of text ( also known as the " snippet " ) . See the options Sites with images : See image thumbnails from the page alongside the snippet for each result . They can help you quickly identify whether the page is relevant to your search term . For example , if you see an image of a furry little critter when you search for [ mouse ] , you can probably deduce that the page is n't talking about computer equipment . Translated foreign pages : Sometimes , the best results for your search are written in another language . Based on your search term , Translate
 d foreign pages automatically chooses the best language ( or languages ) to search in , translates your search , and then translates the results back to the language you prefer to read . Our automatic language selection is n't perfect , of course , so if you 'd like to search specific languages , select them in the panel above the results . Timeline : See results along a timeline , which is especially useful if you 're researching information about a historical person or event . Click any section in the timeline to zoom in on that time period . To change the time range , click Search other dates above the timeline to enter new start and end dates . Alternatively , click the date links below the timeline to see results that mention that particular year , month , or day . Reading level : You can limit your search results to a specific reading level ( Basic , Intermediate , or Advanced ) . Learn more about the Reading level tool . Content type options When you select a specific type of
  content like Videos or Realtime , you 'll often see a set of tools that are unique to that selection . For example , if you select Images , you can filter image results by size , type , and color . Learn more about the search tools specific to Google Images . See definitions for your search term using Dictionary . You can also find synonyms , images , usage examples , and learn if the term is a word in other languages . Discover more results If you 're ever unsure about the precise terms you should use for your search , start out with a broader search term , then use these tools to discover alternative search terms . The Something different option lets you find parallel searches that you might find interesting . See the options Related searches : See related searches at the top of the results page . They 'll stay there , even as you scroll up and down , so you can easily compare and contrast results for different related searches . Wonder wheel : This option visually presents conne
 ctions between related searches and your search term as an interactive diagram . Click the different nodes in the diagram to see how searches can branch out . Something different : The bottom set of links on the side of the search results page includes alternative queries to help you discover webpages that are indirectly related to your search . Learn more about the " Something different " links We hope these options help you explore and interact with your search results in useful ways . As we develop new ways to help you search , you 'll see new or changed options appear . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/122TelsGoog_Answer_1b37_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/122TelsGoog_Answer_1b37_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/122TelsGoog_Answer_1b37_EN.txt.txt
new file mode 100644
index 0000000..41cf76d
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/122TelsGoog_Answer_1b37_EN.txt.txt
@@ -0,0 +1,2 @@
+
+There are several reasons why you may not see dynamic results from Google Instant . Connection speed If you 're searching from a particularly slow Internet connection , you wo n't see results as you type because we do n't want to slow down your search experience . Similarly , if we have historical evidence that the network you 're on tends to be slow , we 'll show you the normal search experience to prevent any possible speed issues . If you 'd prefer to see results as you type anyway , or if you do n't think your connection speed is an issue , you can opt in to seeing dynamic results whenever you 'd like . Just click the Instant link next to the search box , or visit your Preferences page . In addition , if we later identify that your connection becomes stronger , you might start to once again see results as you type . Availability and other issues It 's possible that unexpected issues prevent you from seeing results as you type . For instance , requests from your connection might 
 fail , you might have proxy issues , or your Internet connection could drop mid-search . Try pressing Enter or clicking the " Search " button to see results . If you 're redirected to an error page or other results page , try refreshing the page or going back to the Google homepage . Certain queries If you 're used to seeing results as you type , note that certain types of search queries that could be inappropriate will not trigger automatic results . In these situations , you 'll see a message asking you to click Enter to perform the search . Your preferences and search behavior In addition , you wo n't see results appear as you type if any of the following is true : you search from a page that 's not the Google homepage or a search results page you search from your iGoogle page rather than the classic homepage you search using Google SSL you use an Internet browser that 's not one of these supported browsers : Firefox 3 and later versions , Safari 5 for Mac and later versions , In
 ternet Explorer 8 , and Google Chrome 5 and later versions you search a Google property ( linked to in the side panel ) that does not yet support Google Instant you previously disabled autocomplete you previously opted out of Google Instant you use a Google domain or interface language that does not yet support Google Instant 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/123TelsGoog_Answer_2feb_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/123TelsGoog_Answer_2feb_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/123TelsGoog_Answer_2feb_EN.txt.txt
new file mode 100644
index 0000000..d0cd8aa
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/123TelsGoog_Answer_2feb_EN.txt.txt
@@ -0,0 +1,2 @@
+
+We want our users to feel safe when they search the web , and we 're continuously working to identify dangerous sites and increase protection for our users . This warning message appears with search results we 've identified as sites that may install malicious software on your computer : If you click the title of the result , you 'll be shown the following warning rather than being taken immediately to the webpage in question : You can choose to continue to the site at your own risk by copying and pasting the web address directly in the address bar , then pressing Enter on your keyboard . However , be aware that malicious software is often installed without your knowledge or permission when you visit these sites , and can include programs that delete data on your computer , steal personal information such as passwords and credit card numbers , or alter your search results . For more information on these types of sites , please visit StopBadware . org If one of these sites downloads 
 malicious software onto your computer , please read our additional information about reporting these sites and removing the software from your system . If you 're the administrator of a site we 've identified with this warning message , please visit the instructions found in our Webmaster Help Center to resolve the problem . Note that in some cases , third parties can add malicious code to legitimate sites , which would cause us to show the warning message . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/124TelsGoog_Answer_3024_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/124TelsGoog_Answer_3024_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/124TelsGoog_Answer_3024_EN.txt.txt
new file mode 100644
index 0000000..1fade72
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/124TelsGoog_Answer_3024_EN.txt.txt
@@ -0,0 +1,2 @@
+
+It takes less than a minute to create your own iGoogle homepage , and after that , there are lots of ways to personalize it and really make it your own . 1. Create your page On the Google homepage , click the gear icon and select iGoogle . If this is the first time you 're using iGoogle , you 'll see the setup wizard called " Create your own homepage in under 30 seconds . " Choose some interests , a theme , and a location to get started ( you can add more later ) . Click See your page to create your iGoogle homepage . Use the gear icon to switch between your new iGoogle page and the standard Google homepage called " Classic Home . " You can choose whether or not to sign in to a Google Account when using iGoogle . Learn about the benefits of signing in , including increased privacy and more flexibility in where you can access your iGoogle page . 2. Add more gadgets to your page Gadgets let you use tools , read info , and play games from all across the Web . Here 's how to add a gadge
 t to your page : Click Add gadgets on the right side of the iGoogle page . Gadgets are organized into categories on the left , or you can search on the right . Once you find a gadget you 'd like to add , click Add it now . Click Back to iGoogle home at the upper left side of the page to get back to your page . You can arrange gadgets by dragging and dropping them on the page ( or between tabs ) . Edit your gadgets ' settings by clicking the down-arrow in the title bar and choosing Edit settings . 3. Organize your gadgets into tabs All iGoogle pages start out with a Home tab on the left side of the page . If you have n't added any new tabs , all your gadgets will appear on your Home tab . To add a new tab , follow these steps . Click the downward arrow to the right of your Home tab Select Add a tab . Name the new tab . As you type , you 'll see suggestions for naming your tab , based on names that other users have chosen . Keep the 'I 'm feeling lucky ' box selected , and we 'll auto
 matically add gadgets based on the name of your tab . Click OK . Finally , if you like , you can add a theme to each of your tabs . Watch a video about personalizing your iGoogle page . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/125TelsGoog_Answer_4b5b_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/125TelsGoog_Answer_4b5b_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/125TelsGoog_Answer_4b5b_EN.txt.txt
new file mode 100644
index 0000000..32bea83
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/125TelsGoog_Answer_4b5b_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Tabs let you organize gadgets into groups Tabs appear in the left navigation and let you organize your homepage into different groups of gadgets . For example , you could have a tab for work and a tab for play ( and , yes , the YouTube gadget belongs in 'Play ' ) . Add a tab ... All iGoogle pages start out with a single tab called 'Home , ' but it 's easy to add more . Click the down arrow next to 'Home . ' Click Add a tab . Name the new tab . What 's in a name ? Leave the 'I 'm feeling lucky ' box checked , and we 'll start your new tab off with a few gadgets based on the name you choose . ( Do n't want any help ? Uncheck the box . ) Click OK . ... and put gadgets on it Check the 'I 'm feeling lucky ' box when you name your new tab if you want us to pre-select a few gadgets for you . Add gadgets from the gadget directory ( learn how ) . Drag gadgets from the tab they 're on to the tab you want them to be on . Customize your tabs Tabs can also appear in any order you like , and each
  tab can have its own name , background , and layout style . For instance , you could make a tab named 'Jenny ' with three columns of gadgets on a classic background , or a tab called 'Games ' with two medium-sized columns and the 'Theme of the day , ' or ... well , we could go on and on ... and we do , on the iGoogle Preferences page . Click the link or follow these steps to get there : Click the name of the tab you want to edit . Click the down arrow next to the name . Click Edit this tab . Be sure to Save any changes you make on the Preferences page . Share a tab with a friend Think you 've made the perfect tab ? You can send it to others . Click the tab you want to share . Click the down arrow next to the name . Click Share this tab . Check the boxes for the gadgets you 'd like to send , enter your friend 's email address , and write a message . If you check the 'Send my settings for these gadgets ' box , the gadgets will appear to your friend much as they appear to you . But sh
 aring works a little differently for every gadget , so be careful when sharing tabs or gadgets that include personal info . Click Send . Delete a tab Click the tab you want to delete . Click the down arrow next to its name . Click Delete this tab . Confirm by clicking Delete this tab again . ( Remember : deleting a tab deletes all the gadgets on it . Keep a gadget by dragging it to another tab . ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/126TelsGoog_Answer_596f_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/126TelsGoog_Answer_596f_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/126TelsGoog_Answer_596f_EN.txt.txt
new file mode 100644
index 0000000..133b37d
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/126TelsGoog_Answer_596f_EN.txt.txt
@@ -0,0 +1,2 @@
+
+When you use Google to search , we try to provide the best possible results . To do that , we sometimes customize your search results based on one or more factors . Click View customizations , which may appear at the bottom of the search results page or to the right of the search box , to see what type of information was used to customize your results . ( To protect your privacy , this information becomes unavailable after a brief period of time . Learn more about Google 's privacy policy ) Types of customizations Search history Sometimes , we customize your search results based on your past search activity on Google , such as searches you 've done or results you 've clicked . If you 're signed in to your Google Account and have Web History enabled , these customizations are based on your Web History . If you 're signed in and do n't have Web History enabled , no search history customizations will be made . ( Using Web History , you can control exactly what searches are stored and u
 sed to personalize your results . Learn about using Web History ) If you are n't signed in to a Google Account , your search results may be customized based on past search information linked to your browser using a cookie . Because many people might be searching on one computer , Google does n't show a list of previous search activity on this computer . Learn how to turn off these customizations Location We try to use information about your location to customize your search results if there 's a reason to believe it 'll be helpful ( for example , if you search for a restaurant chain , you may want to find the one near you ) . That customization may rely on a default location that you 've previously specified ( for example , in Google Maps ) or an approximate location based on your IP address . Learn more about location 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/127TelsGoog_Answer_5b4c_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/127TelsGoog_Answer_5b4c_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/127TelsGoog_Answer_5b4c_EN.txt.txt
new file mode 100644
index 0000000..d6b6b43
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/127TelsGoog_Answer_5b4c_EN.txt.txt
@@ -0,0 +1,2 @@
+
+The Basic search help article covers all the most common issues , but sometimes you need a little bit more power . This document will highlight the more advanced features of Google Web Search . Have in mind though that even very advanced searchers , such as the members of the search group at Google , use these features less than 5 % of the time . Basic simple search is often enough . As always , we use square brackets [ ] to denote queries , so [ to be or not to be ] is an example of a query ; [ to be ] or [ not to be ] are two examples of queries . Phrase search ( " " ) By putting double quotes around a set of words , you are telling Google to consider the exact words in that exact order without any change . Google already uses the order and the fact that the words are together as a very strong signal and will stray from it only for a good reason , so quotes are usually unnecessary . By insisting on phrase search you might be missing good results accidentally . For example , a sear
 ch for [ " Alexander Bell " ] ( with quotes ) will miss the pages that refer to Alexander G. Bell . Search within a specific website ( site : ) Google allows you to specify that your search results must come from a given website . For example , the query [ iraq site : nytimes . com ] will return pages about Iraq but only from nytimes . com . The simpler queries [ iraq nytimes . com ] or [ iraq New York Times ] will usually be just as good , though they might return results from other sites that mention the New York Times . You can also specify a whole class of sites , for example [ iraq site : . gov ] will return results only from a . gov domain and [ iraq site : . iq ] will return results only from Iraqi sites . Terms you want to exclude ( - ) Attaching a minus sign immediately before a word indicates that you do not want pages that contain this word to appear in your results . The minus sign should appear immediately before the word and should be preceded with a space . For exampl
 e , in the query [ anti-virus software ] , the minus sign is used as a hyphen and will not be interpreted as an exclusion symbol ; whereas the query [ anti-virus -software ] will search for the words 'anti-virus ' but exclude references to software . You can exclude as many words as you want by using the - sign in front of all of them , for example [ jaguar -cars -football -os ] . The - sign can be used to exclude more than just words . For example , place a hyphen before the 'site : ' operator ( without a space ) to exclude a specific site from your search results . Fill in the blanks ( * ) The * , or wildcard , is a little-known feature that can be very powerful . If you include * within a query , it tells Google to try to treat the star as a placeholder for any unknown term(s ) and then find the best matches . For example , the search [ Google * ] will give you results about many of Google 's products ( go to next page and next page -- we have many products ) . The query [ Obama 
 voted * on the * bill ] will give you stories about different votes on different bills . Note that the * operator works only on whole words , not parts of words . Search exactly as is ( + ) Google employs synonyms automatically , so that it finds pages that mention , for example , childcare for the query [ child care ] ( with a space ) , or California history for the query [ ca history ] . But sometimes Google helps out a little too much and gives you a synonym when you do n't really want it . By attaching a + immediately before a word ( remember , do n't add a space after the + ) , you are telling Google to match that word precisely as you typed it . Putting double quotes around a single word will do the same thing . The OR operator Google 's default behavior is to consider all the words in a search . If you want to specifically allow either one of several words , you can use the OR operator ( note that you have to type 'OR ' in ALL CAPS ) . For example , [ San Francisco Giants 200
 4 OR 2005 ] will give you results about either one of these years , whereas [ San Francisco Giants 2004 2005 ] ( without the OR ) will show pages that include both years on the same page . The symbol | can be substituted for OR . ( The AND operator , by the way , is the default , so it is not needed . ) Exceptions Search is rarely absolute . Search engines use a variety of techniques to imitate how people think and to approximate their behavior . As a result , most rules have exceptions . For example , the query [ for better or for worse ] will not be interpreted by Google as an OR query , but as a phrase that matches a ( very popular ) comic strip . Google will show calculator results for the query [ 34 * 87 ] rather than use the 'Fill in the blanks ' operator . Both cases follow the obvious intent of the query . Here is a list of exceptions to some of the rules and guidelines that were mentioned in this and the Basic Search Help article : Exceptions to 'Every word matters ' Words 
 that are commonly used , like 'the , ' 'a , ' and 'for , ' are usually ignored ( these are called stop words ) . But there are even exceptions to this exception . The search [ the who ] likely refers to the band ; the query [ who ] probably refers to the World Health Organization -- Google will not ignore the word 'the ' in the first query . Synonyms might replace some words in your original query . ( Adding + before a word disables synonyms . ) A particular word might not appear on a page in your results if there is sufficient other evidence that the page is relevant . The evidence might come from language analysis that Google has done or many other sources . For example , the query [ overhead view of the bellagio pool ] will give you nice overhead pictures from pages that do not include the word 'overhead . ' Punctuation that is not ignored Punctuation in popular terms that have particular meanings , like [ C++ ] or [ C# ] ( both are names of programming languages ) , are not igno
 red . The dollar sign ( $ ) is used to indicate prices . [ nikon 400 ] and [ nikon $400 ] will give different results . The hyphen - is sometimes used as a signal that the two words around it are very strongly connected . ( Unless there is no space after the - and a space before it , in which case it is a negative sign . ) The underscore symbol _ is not ignored when it connects two words , e. g. [ quick_sort ] . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/128TelsGoog_Answer_70d7_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/128TelsGoog_Answer_70d7_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/128TelsGoog_Answer_70d7_EN.txt.txt
new file mode 100644
index 0000000..c0731b8
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/128TelsGoog_Answer_70d7_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Welcome to the Help Center ! Take a minute to learn about the resources at your disposal , so you can get to the information you need as quickly as possible . To add some fun , Matt from the Google team also created a musical tour of the Help Center that you can watch : Search for your answer . Type your question in the search box at the top of the Help Center , then click the Search Help button to see relevant articles and forum discussions . Browse recommended and popular articles . . You can find these articles on the right side of the page . These recommendations are based on your Help Center searches and articles that many other people have viewed . Check out key help resources . We 've also called out some special help resources -- such as comprehensive guides , video tutorials , and lists of known product issues -- on the right side of the page . Use links on the left to navigate . Links to various topics of the Help Center are listed on the left , to help you quickly get to 
 different sections in the site . You can also click the top Help articles link to go to the Help Center homepage . Ask questions in the Help Forum . Some Help Centers have a Help Forum section where you can share tips and ask other users questions about the product . Need more help ? Find out how to contact us . Click the Contacting us link at the bottom of the page to learn about options available for contacting the Google team . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/129TelsGoog_Answer_770f_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/129TelsGoog_Answer_770f_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/129TelsGoog_Answer_770f_EN.txt.txt
new file mode 100644
index 0000000..e64c64d
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/129TelsGoog_Answer_770f_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Google will always provide the results that we believe are most relevant to your search . Location is one of a number of factors that we use to provide these relevant results . For example , if you search for a restaurant chain , you may want to find one near you . When searching for movie showtimes , you may want listings in your area . Such search results are made possible by location-based customization . The location used to customize your results is clearly indicated on the left side of your results page . Our location detection technology will automatically set a location for you , or you can easily specify an alternate location . Automatically detected location To provide you with the most relevant results , Google attempts to automatically detect your location and customize results based on that detected location . A location that 's labeled " Auto-detected " is chosen based on the following factors : Your IP address . Google Toolbar 's My Location feature . If you have Goog
 le Toolbar installed and have the My Location feature enabled , your approximate location ( if detected ) will be used to customize your search results . If you do n't want Toolbar to attempt to determine your approximate location , you can turn off My Location See the wrong location ? We 're continuing to improve our location detection technology , and the automatically detected location may be less accurate in certain areas . If you notice an incorrect location , please set your location manually by following the instructions below . If you 're being redirected to a Google domain that does n't match the country you 're in ( for example , Google . com . au instead of Google . com ) , please let us know ; we may be incorrectly detecting your IP address . Do n't see a location ? Sometimes we wo n't show any location if we ca n't accurately determine one for you . We 're working hard to improve our location detection technology . Also , the location setting will not always be availabl
 e depending on your language and region . We are working to make the location setting more widely available . In the meantime , if you do n't have any location shown , it 's easy to set a location yourself by following the instructions below . Specify your preferred location If you 'd like to change your location , click Change Location and enter a street address , zip code , city and state , or country in the box that appears . Click Set to save your location setting . You can only specify a location within the country of your current Google domain . For instance , it 's not possible to set a U. S. address on www . google . es , the Google domain for Spain . If you 'd like to see results for a region outside of your Google domain , please specify the location in the search query ( like [ bicycle repair paris ] ) or visit a different Google local domain instead . If you don\u2019t already have a location set , enter a location in the box next to " Search near , " then click Set . Your 
 manually set location will be saved in a browser cookie on your specific computer and internet browser only . When you use the same computer and browser , this manually set location will be used to customize your search results instead of the automatically detected location . If your location preference doesn\u2019t remain saved on the browser or computer where you set it , please check your browser settings . To clear your manually set location , click on the auto-detected location . This will clear the location setting saved in your browser cookie , and your results will be customized based on the automatically detected location . Can I turn off location-based customization ? The customization of search results based on location is an important component of a consistent , high-quality search experience . Therefore , we have n't provided a way to turn off location customization , although we 've made it easy for you to set your own location or to customize using a general location as 
 broad as the country that matches your local domain . If you find that your results for a particular search are more local than what you 're looking for , you can set your location to a broader geographical area ( such as a country instead of a city , zip code , or street address ) . Please note that this will greatly reduce the amount of locally relevant results that you\u2019ll see . What information about location does Google store ? Google has many services that collect or use location data , including Google Maps , Latitude , Toolbar , and others . If you have questions about our privacy policy , please refer to our privacy center . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/130TelsGoog_Answer_91e4_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/130TelsGoog_Answer_91e4_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/130TelsGoog_Answer_91e4_EN.txt.txt
new file mode 100644
index 0000000..53847d1
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/130TelsGoog_Answer_91e4_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Some people have reported difficulty saving settings on the Google Preferences page , such as the following : The level of SafeSearch filtering The language used to display Google and your search results The way your search results are displayed Issues with your search settings are likely caused by problems with your browser cookies . Cookies are small files on your computer that contain information about your settings for specific websites . Try the following actions to clean up your cookies . After each step , see if the problem 's been resolved . Confirm that cookies are enabled on your browser . If your browser does n't allow cookies to be stored , your preferences ca n't be retrieved properly when you visit Google . See browser-specific instructions for enabling cookies . After confirming that cookies are enabled , verify your search settings on the Google Preferences page . Check settings on your computer . Many types of software can hamper your browser 's ability to save your
  cookies and remember your settings . If you use any of the following kinds of programs , you may want to inspect how they handle cookies : Firewall , proxy , or anti-virus programs are often configured to delete cookies on a regular basis . Internet filtering software , such as programs designed to restrict underage access to adult sites , may be responsible for overriding your SafeSearch preferences . Browser modes designed for private browsing ( like incognito mode on Google Chrome ) might not allow your browser to store your settings in cookies . Try disabling this browsing mode . Clear your browser cache and cookies . This step removes your settings for all websites , but also ensures that corrupted browser files are deleted . See browser-specific instructions for deleting these files . After removing these files , go back to the Google Preferences page to reset your preferences . If you continue to experience issues , visit the Web Search Help Forum . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/131TelsGoog_Answer_94d0_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/131TelsGoog_Answer_94d0_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/131TelsGoog_Answer_94d0_EN.txt.txt
new file mode 100644
index 0000000..9810579
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/131TelsGoog_Answer_94d0_EN.txt.txt
@@ -0,0 +1,2 @@
+
+In a nutshell Most gadgets are created by third parties . Contact info is usually located on the gadget 's homepage . The simplest step may be to remove the gadget from your homepage . Learn how . Non-Google gadgets Most of the gadgets for your iGoogle page are created and maintained by third-party developers . If you have questions or concerns about the functionality or content of a particular gadget , your best bet is to try to contact the gadget creator directly . Click the down arrow in the top corner of the gadget . Click About this gadget . Do n't have the gadget on your page anymore ? Search for it in the gadget directory and click its name ( do n't click 'Add it now ' ) . The author of the gadget is listed to the right of the gadget . Click the author name to see more details . On the following page , click the Contact link beneath the author 's name . Google gadgets For problems with gadgets related to a specific Google product , such as the Gmail gadget or Google Calendar 
 gadget , the best first step is to try the Google service on its main site ( for instance , if your Gmail gadget is n't working right , try visiting mail . google . com to make sure Gmail is still working ) . If it 's just the gadget , let us know by posting in the Web Search Help Forum . Copyright concerns If you are a copyright holder and want to request removal of material that you believe infringes your copyright , see our DMCA notification instructions . Policy concerns If you want to report a gadget that 's violating our Program Policy , contact us . Note that we 'll only take action on your report if the gadget violates our policies , so review the the Program Policy first . Report gadgets that break iGoogle or violate other program policies 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/132TelsGoog_Answer_a67e_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/132TelsGoog_Answer_a67e_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/132TelsGoog_Answer_a67e_EN.txt.txt
new file mode 100644
index 0000000..429fe4e
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tels/132TelsGoog_Answer_a67e_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Common symptoms of malware If you 're seeing suspicious pop-ups , unwanted toolbars , redirects , strange Google search results , or other unexpected behavior on your computer , you may have been tricked into installing malicious software ( also known as 'malware ' ) on your computer . In this article , we 've provided some recommendations on how you can prevent and remove malware . ( Note that Google is n't affiliated with any of the programs we suggest in this article , and we 're unable to vouch for their effectiveness . Our users have told us they often work . ) If clicking a Google search result has redirected you to a suspicious site , please report the suspicious site before trying the anti-spyware software below . This will help us investigate malware activity and hopefully help prevent it in the future . First , use anti-spyware software to detect and remove malware from your computer The following programs can remove the most common malware . You may need to try more than 
 one to catch every instance of malware on your computer . Make sure to use the latest versions of these programs . Malwarebytes ' Anti-Malware Spyware Doctor with Anti-Virus , which is available as a part of Google Pack MacScan ( Mac ) Check to see if the issue has been resolved . Still having trouble ? Try these more advanced techniques READ THIS FIRST : These techniques involve some advanced steps , like reviewing and analyzing logs produced by your computer . If you 're not sure how to take these steps properly , you may want to consult a technician . Deleting legitimate lines in your HijackThis log can have serious repercussions for your computer . Use a bootable anti-virus rescue CD to scan your computer If the malware is preventing your computer from starting properly , you may want to use a bootable anti-virus CD to scan and clean your files . This method of scanning your computer helps detect malware that may be deeply embedded within your system files . Such malware may go 
 undetected when using normal anti-virus detection methods . Use HijackThis to detect changed settings ( Windows only ) Many people have used HijackThis , an advanced anti-spyware application , to find settings on their computers that may have been changed by malware . HijackThis will create a log of files and settings that you should then analyze , to determine if they have been affected by malware . Here are some support forums where you can post your HijackThis log and receive technical assistance . Google is n't affiliated with these forums , and we ca n't make any promises about the quality of the advice they might provide . After using HijackThis , run anti-spyware programs again , such as the ones listed above , to see if additional malware shows up . Turn off System Restore and scan again ( Windows only ) System Restore is a Windows backup feature ; it periodically backs up files on your computer in case you need to revert to an earlier version of your operating system at a l
 ater point . If the malware was inadvertently included in a system restore point , you 'll need to disable System Restore , before anti-spyware programs can remove the malware . Here 's how : Clear your browser 's cache . See instructions Disable the System Restore functionality on your computer ( administrative rights required ) . Go to the Start menu . Select Control Panel . Double-click System . Click the System Restore tab . Select the 'Turn off System Restore ' checkbox . Restart your computer . Run anti-spyware programs again to see if additional malware shows up . Once you 're done checking your system files for malware , remember to re-enable System Restore . If you continue to experience symptoms of malware at this point , consult a technician for additional help . Practice safe browsing to avoid malware To prevent malware from infiltrating your computer in the future , follow these general best practices : Upgrade your computer to use the latest operating systems and softw
 are . Turn on automatic updates for your operating system . Turn on automatic updates for your programs . Regularly check for updates for programs that do n't have automatic update functionality , such as Adobe Flash . Do n't click on links you do n't know the destination of . Do n't download email attachments from anyone you do n't know . Sign in to user accounts with limited permissions on your computer . This lessens the ability of malware to make system-wide changes to your computer . Install a firewall and antivirus software , ensuring automatic updates are enabled . If you use an email program on your computer to send and receive messages , install spam filtering software . Many email programs also come with built-in spam filters . Report malware If you feel you were deceived when you installed a program that creates popups or modifies your browser , you may want to file a complaint at StopBadware . org . Additionally , you may want to contact the Federal Trade Commission ( FT
 C ) , which handles complaints about deceptive or unfair business practices in the U. S. To file a complaint , do one of the following : Visit https://www.ftccomplaintassistant.gov/ Call 1-877-FTC-HELP Write to : Federal Trade Commission , CRC-240 , Washington , D. C. 20580 If your complaint is against a company that 's outside the United States , visit http://www.econsumer.gov 
\ No newline at end of file


[07/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/177ted_william_li.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/177ted_william_li.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/177ted_william_li.txt
new file mode 100644
index 0000000..89e6353
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/177ted_william_li.txt
@@ -0,0 +1,2 @@
+
+Good afternoon . There 's a medical revolution happening all around us , and it 's one that 's going to help us conquer some of society 's most dreaded conditions , including cancer . And the revolution is called angiogenesis , and it 's based on the process that our bodies use to grow blood vessels . So why should we care about blood vessels ? Well , the human body is literally packed with them , 60,000 miles worth in a typical adult . End to end , that would form a line that would circle the earth twice . The smallest blood vessels are called capillaries . We 've got 19 billion of them in our bodies . And these are the vessels of life , and , as I 'll show you , they can also be the vessels of death . Now the remarkable thing about blood vessels is that they have this ability to adapt to whatever environment they 're growing in . For example , in the liver they form channels to detoxify the blood . In the lung , they line air sacs for gas exchange . In muscle , they corkscrew so t
 hat muscles can contract without cutting off circulation . And in nerves , they course along like power lines , keeping those nerves alive . And we get most of these blood vessels when we 're actually still in the womb . And what that means is that , as adults , blood vessels do n't normally grow , except in a few special circumstances . In women , blood vessels grow every month to build the lining of the uterus . During pregnancy , they form the placenta , which connects mom and baby . And after injury , blood vessels actually have to grow under the scab in order to heal a wound . And this is actually what it looks like . Hundreds of blood vessels all growing to the center of the wound . So the body has the ability to regulate the amount of blood vessels that are present at any given time . And it does this through an elaborate and elegant system of checks and balances , stimulators and inhibitors of angiogenesis , such that , when we need a brief burst of blood vessels , the body 
 can do this by releasing stimulators , proteins called angiogenic factors that act as natural fertilizer and stimulate new blood vessels to sprout . And when those excess vessels are no longer needed , the body prunes them back to baseline using naturally occurring inhibitors of angiogenesis . Now there are other situations where we start beneath the baseline , and we need to grow more blood vessels just to get back to normal levels . For example , after an injury . And a body can do that too , but only to that normal level , that set point . But what we now know is , for a number of diseases , there are defects in the system , where the body ca n't prune back extra blood vessels or ca n't grow enough new ones in the right place at the right time . And in these situations , angiogenesis is out of balance . And when angiogenesis is out of balance , a myriad of diseases result . For example , insufficient angiogenesis , not enough blood vessels , leads to wounds that do n't heal , hea
 rt attacks , legs without circulation , death from stroke , nerve damage . And on the other end , excessive angiogenesis , too many blood vessels , drives disease . And we see this in cancer , blindness , arthritis , obesity , Alzheimer 's disease . In total , there are more than 70 major diseases , affecting more than a billion people worldwide , that all look on the surface to be different from one another , but all actually share abnormal angiogenesis as their common denominator . And this realization is allowing us to reconceptualize the way that we actually approach these diseases by controlling angiogenesis . Now I 'm going to focus on cancer because angiogenesis is a hallmark of cancer , every type of cancer . So here we go . This is a tumor , dark , gray , ominous mass growing inside a brain . And under the microscope , you can see hundreds of these brown staining blood vessels , capillaries that are feeding cancer cells , bringing oxygen and nutrients . But cancers do n't s
 tart out like this . And , in fact , cancers do n't start out with a blood supply . They start out as small , microscopic nests of cells That can only grow to one half a cubic millimeter in size . That 's the tip of a ballpoint pen . Then they ca n't get any larger because they do n't have a blood supply , so they do n't have enough oxygen or nutrients . And in fact , we 're probably forming these microscopic cancers all the time in our body . Autopsy studies from people who died in car accidents have shown that about 40 percent of women between the ages of 40 and 50 actually have microscopic cancers in their breasts . About 50 percent of men in their 50s and 60s have microscopic prostate cancers . And virtually 100 percent of us , by the time we reach our 70s , will have microscopic cancers growing in our thyroid . Yet , without a blood supply , most of these cancers will never become dangerous . Dr. Judah Folkman , who was my mentor , and who was the pioneer of the angiogenesis fi
 eld , once called this " cancer without disease . " So the body 's ability to balance angiogenesis , when it 's working properly , prevents blood vessels from feeding cancers . And this turns out to be one of our most important defense mechanisms against cancer . In fact , if you actually block angiogenesis and prevent blood vessels from ever reaching cancer cells , tumors simply ca n't grow up . But once angiogenesis occurs , cancers can grow exponentially . And this is actually how a cancer goes from being harmless to deadly . Cancer cells mutate and they gain the ability to release lots of those angiogenic factors , natural fertilizer , that tip the balance in favor of blood vessels invading the cancer . And once those vessels invade the cancer , it can expand , it can invade local tissues . And the same vessels that are feeding tumors , allow cancer cells to exit into the circulation as metastases . And , unfortunately , this late stage of cancer is the one at which it 's most l
 ikely to be diagnosed , when angiogenesis is already turned on , and cancer cells are growing like wild . So , if angiogenesis is a tipping point between a harmless cancer and a harmful one , then one major part of the angiogenesis revolution is a new approach to treating cancer by cutting off the blood supply . We call this antiangiogenic therapy , and it 's completely different from chemotherapy because it selectively aims at the blood vessels that are feeding the cancers . And we can do this because tumor blood vessels are unlike normal , healthy vessels we see in other places of the body . They 're abnormal ; they 're very poorly constructed ; and , because of that , they 're highly vulnerable to treatments that target them . In effect , when we give cancer patients antiangiogenic therapy -- here , an experimental drug for a glioma , which is a type of brain tumor -- you can see that there are dramatic changes that occur when the tumor is being starved . Here 's a woman with a b
 reast cancer being treated with the antiangiogenic drug called Avastin , which is FDA approved . And you can see that the halo of blood flow disappears after treatment . Well , I 've just shown you two very different types of cancer that both responded to antiangiogenic therapy . So , a few years ago , I asked myself , " Can we take this one step further , and treat other cancers , even in other species ? " So here is a nine year-old boxer named Milo who had a very agressive tumor called a malignant neurofibroma growing on his shoulder . It invaded into his lungs . His veterinarian only gave him three months to live . So we created a cocktail of antiangiogenic drugs that could be mixed into his dog food as well as an antiangiogenic cream that could be applied on the surface of the tumor . And within a few weeks of treatment , we were able to slow down that cancer 's growth such that we were ultimately able to extend milo 's survival to six times what the veterinarian had initially p
 redicted , all with a very good quality of life . And we subsequently treated more than 600 dogs . We have about a 60 percent response rate and improved survival for these pets that were about to be euthanized . So let me show you a couple of even more interesting examples . This is 20 year old dolphin living in Florida , and she had these lesions in her mouth that , over the course of three years , developed into invasive squamous cell cancers . So we created an antiangiogenic paste . We had it painted on top of the cancer three times a week . And over the course of seven months , the cancers completely disappeared , and the biopsies came back as normal . Here 's a cancer growing on the lip of a quarter horse named Guiness . It 's a very , very deadly type of cancer called an angiosarcoma . It had already spread to his lymph nodes , so we used an antiangiogenic skin cream for the lip and an oral cocktail , so we could treat from the inside as well as the outside . And over the cour
 se of six months , he experienced a complete remission . And here he is six years later , Guiness , with his very happy owner . ( Applause ) Now , obviously , antiangiogenic therapy could be used for a wide range of cancers . And , in fact , the first pioneering treatments , for people , as well as dogs , are already becoming available . There 's 12 different drugs , 11 different cancer types , but the real question is : How well do these work in practice ? So here 's actually the patient survival data from eight different types of cancer . And the bars represent survival time taken from the era in which there was only chemotherapy , or surgery , or radiation available . But starting in 2004 , when antiangiogenic therapies first became available , well you can see that there has been a 70 to 100 percent improvement in survival for people with kidney cancer , multiple myeloma , colorectal cancer , and gastrointestinal stromal tumors . That 's impressive . But for other tumors and can
 cer types , the improvements have only been modest . So I started asking myself , " Why have n't we been able to do better ? " And the answer , to me , is obvious ; we 're treating cancer too late in the game , when it 's already established , and , oftentimes , it 's already spread or metastasized . And as a doctor , I know that , once a disease progresses to an advanced stage , achieving a cure can be difficult , if not impossible . So I went back to the biology of angiogenesis and started thinking : Could the answer to cancer be preventing angiogenesis , beating cancer at its own game so the cancers could never become dangerous ? This could help healthy people as well as people who 've already beaten cancer once or twice and want to find a way to keep it from coming back . So to look for a way to prevent angiogenesis in cancer , I went back to look at cancer 's causes . And what really intrigued me was when I saw that diet accounts for 30 to 35 percent of environmentally caused c
 ancers . Now , the obvious thing is to think about what we could remove from our diet , what what to strip out , take away . But I actually took a completely opposite approach and began asking : What could we be adding to our diet that 's naturally antiangiogenic , that could boost the body 's defense system and beat back those blood vessels that are feeding cancers ? In other words , can we eat to starve cancer ? Well , the answer 's yes . And I 'm going to show you how . And our search for this has taken us to the market , the farm and to the spice cabinet because what we 've discovered is that mother nature has laced a large number of foods and beverages and herbs with naturally occurring inhibitors of angiogenesis . So here 's a test system we developed . At the center is a ring from which hundreds of blood vessels are growing out in a star burst fashion . And we can use this system to test dietary factors at concentrations that are obtainable by eating . So let me show you what
  happens when we put in an extract from red grapes . The active ingredient 's resveratrol . It 's also found in red wine . This inhibits abnormal angiogenesis by 60 percent . Here 's what happens when we add an extract from strawberries . It potently inhibits angiogenesis . And extract from soy beans . And here is a growing list of our antiangiogenic foods and beverages that we 're interested in studying . And for each food type , we believe there is different potencies within different strains and varietals . And we want to measure this because , well , while you 're eating a strawberry or drinking tea , why not select the one that 's most potent for preventing cancer . So here are four different teas that we 've tested . They 're all common ones , Chinese jasmine , Japanese sencha , Earl Grey and a special blend that we prepared . And you can see clearly that the teas vary in their potency from less potent to more potent . But what 's very cool is when we actually combined the two
  less potent teas together , the combination , the blend , is more potent than either one alone . This means there 's food synergy . Here 's some more data from our testing . Now , in the lab , we can simulate tumor angiogenesis represented here in a black bar . And using this system , we can test the potency of cancer drugs . So the shorter the bar , less angiogenesis , that 's good . And here are some common drugs that have been associated with reducing the risk of cancer in people . Statins , nonsteroidal anti-inflammatory drugs and a few others , they inhibit angiogenesis too . And here are the dietary factors going head to head against these drugs . You can see , they clearly hold their own and , in some cases , they 're more potent than the actual drugs . Soy , parsley , garlic , grapes , berries , I could go home and cook a tasty meal using these ingredients . So imagine if we could create the world 's first rating system in which we could score foods according to their antia
 ngiogenic cancer-preventative properties . And that 's what we 're doing right now . Now , I 've shown you a bunch of lab data , and so the real question is : What is the evidence in people that eating certain foods can reduce angiogenesis in cancer ? Well , the best example I know is a study of 79,000 men , followed over 20 years , in which it was found that men whom consumed cooked tomatoes two to three times a week had up to a 50 percent reduction in their risk of developing prostate cancer . Now , we know that tomatoes are a good source of lycopene , and lycopene is antiangiogenic . But what 's even more interesting from this study is that those men who did develop prostate cancer , those who ate more servings of tomato sauce actually had fewer blood vessels feeding their cancer . So this human study is a prime example of how antiangiogenic substances present in food and consumed at practical levels can impact on cancer . And we 're now studying the role of a healthy diet with D
 ean Ornish at UCSF and Tufts University on the role of this healthy diet on markers of angiogenesis that we can find in the bloodstream . Now , obviously , what I 've shared with you has some far-ranging implications even beyond cancer research . Because if we 're right , it could impact on consumer education , food services , public health and even the insurance industry . And , in fact , some insurance companies are already beginning to think along these lines . Check out this ad from Blue Cross Blue Shield of Minnesota . And for many people around the world , dietary cancer prevention may be the only practical solution because not everybody can afford expensive end-stage cancer treatments , but everybody could benefit from a healthy diet based on local , sustainable , antiangiogenic crops . Now , finally , I 've talked to you about food , and I 've talked to you about cancer , so there 's just one more disease that I have to tell you about and that 's obesity . Because it turns o
 ut that adipose tissue , fat , is highly angiogenesis dependent . And , like a tumor , fat grows when blood vessels grow . So the question is : Can we shrink fat by cutting off its blood supply ? So the top curve shows the body weight of a genetically obese mouse that eats nonstop , until it turns fat like this furry tennis ball . And the bottom curve is the weight of a normal mouse . If you take the obese mouse and give it an angiogenesis inhibitor , it loses weight . Stop the treatment , gains the weight back . Restart the treatment , loses the weight again . Stop the treatment , it gains the weight back . And , in fact , you can cycle the weight up and down simply by inhibiting angiogenesis . So this approach that we 're taking for cancer prevention may also have an application for obesity . The really , truly interesting thing about this is that we ca n't take these obese mice and make them lose more weight than what the normal mouse 's weight is supposed to be . In other words 
 , we ca n't create supermodel mice . ( Laughter ) And this speaks to the role of angiogenesis in regulating healthy set points . Albert Szent-Gyorgi once said that , " Discovery consists of seeing what everyone has seen , and thinking what no one has thought . " I hope I 've convinced you that , for diseases like cancer , obesity and other conditions , that there may be a great power in attacking their common denominator , angiogenesis . And that 's what I think the world needs now . Thank you . ( Applause ) June Cohen : So these drugs are n't exactly -- they 're not exactly in mainstream cancer treatments right now . For anyone out here who has cancer , what would you recommend ? Do you recommend pursuing these treatments now , for most cancer patients ? William Li : So there are antiangiogenic treatments that are FDA approved . And if you 're a cancer patient or working for one or advocating for one , you should ask about them . And there are many clinical trials . The Angiogenesi
 s Foundation is following almost 300 companies , and there 's about 100 more drugs in that pipeline . So consider the approved ones , look for clinical trials , but then between what the doctor can do for you , we need to start asking what can we do for ourselves . And this is one of the themes that I 'm talking about is we can empower ourselves to do the things that doctors ca n't do for us , which is to use knowledge and take action . And if mother nature has given us some clues , we think that there might be a new future in the value of how we eat . And what we eat is really our chemotherapy three times a day . JC : Right . And along those lines , for people who might have risk factors for cancer , would you recommend pursuing any treatments sort of prophylactically or simply pursuing the right diet with lots of tomato sauce ? WL : Well , you know , there 's an abundant epidemiological evidence . And I think in the information age , it does n't take long to go to a credible sourc
 e like Pubmed , the National Library of Medicine , to look for epidemiological studies for cancer risk reduction based on diet and based on common medications . And that 's certainly something that anybody can look into . JC : Okay . Well , thank you so much . ( Applause ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/tkhv/297tkhv\" url=\"http:__www.company-wizard.co.uk_SEO_Link-Exchange.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/tlg\"/250tlg\" url=\"http:__www.ourpartnership.org.uk_newspub_story.cfm?id=353&sid=123.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/togq/298togq\" url=\"http:__www.hearingconcern.org.uk_news_news_scots06.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/tqru/299tqru\" url=\"http:__histru.bournemouth.ac.uk_Oral_History_Technology_in_the_Community_Joan_Wolstenholme.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/uclj/300uclj\" url=\"http:__beehive.thisislincolnshire.co.uk_default.asp?WCI=SiteHome&ID=4337&PageID=21591.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/uijm/301uijm\" url=\"http:__www.galha.freeserve.co.uk_glh_233_mckenna.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/A/178un_A_AC252_L13.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/un/A/178un_A_AC252_L13.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/A/178un_A_AC252_L13.txt
new file mode 100644
index 0000000..3d942b2
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/A/178un_A_AC252_L13.txt
@@ -0,0 +1,2 @@
+
+Ad Hoc Committee established by General Assembly resolution 51/210 of 17 December 1996 Eighth session 28 June-2 July 2004 Consolidated text prepared by the Bureau of the Committee Draft International Convention for the Suppression of Acts of Nuclear Terrorism The States Parties to this Convention , Having in mind the purposes and principles of the Charter of the United Nations concerning the maintenance of international peace and security and the promotion of good-neighbourliness and friendly relations and cooperation among States , Recalling the Declaration on the Occasion of the Fiftieth Anniversary of the United Nations of 24 October 1995 , Recognizing the right of all States to develop and apply nuclear energy for peaceful purposes and their legitimate interests in the potential benefits to be derived from the peaceful application of nuclear energy , Bearing in mind the Convention on the Physical Protection of Nuclear Material of 1980 , Deeply concerned about the worldwide escal
 ation of acts of terrorism in all its forms and manifestations , Recalling also the Declaration on Measures to Eliminate International Terrorism , annexed to General Assembly resolution 49/60 of 9 December 1994 , in which , inter alia , the States Members of the United Nations solemnly reaffirm their unequivocal condemnation of all acts , methods and practices of terrorism as criminal and unjustifiable , wherever and by whomever committed , including those which jeopardize the friendly relations among States and peoples and threaten the territorial integrity and security of States , Noting that the Declaration also encouraged States to review urgently the scope of the existing international legal provisions on the prevention , repression and elimination of terrorism in all its forms and manifestations , with the aim of ensuring that there is a comprehensive legal framework covering all aspects of the matter , Recalling General Assembly resolution 51/210 of 17 December 1996 and the D
 eclaration to Supplement the 1994 Declaration on Measures to Eliminate International Terrorism annexed thereto , Recalling also that , pursuant to General Assembly resolution 51/210 , an ad hoc committee was established to elaborate , inter alia , an international convention for the suppression of acts of nuclear terrorism to supplement related existing international instruments , Noting that acts of nuclear terrorism may result in the gravest consequences and may pose a threat to international peace and security , Noting also that existing multilateral legal provisions do not adequately address those attacks , Being convinced of the urgent need to enhance international cooperation between States in devising and adopting effective and practical measures for the prevention of such acts of terrorism and for the prosecution and punishment of their perpetrators , Noting that the activities of military forces of States are governed by rules of international law outside of the framework o
 f this Convention and that the exclusion of certain actions from the coverage of this Convention does not condone or make lawful otherwise unlawful acts , or preclude prosecution under other laws , Have agreed as follows : Article 1 For the purposes of this Convention : \u201c Radioactive material \u201d means nuclear material and other radioactive substances which contain nuclides which undergo spontaneous disintegration ( a process accompanied by emission of one or more types of ionizing radiation , such as alpha- , beta- , neutron particles and gamma rays ) and which may , owing to their radiological or fissile properties , cause death , serious bodily injury or substantial damage to property or to the environment . \u201c Nuclear material \u201d means plutonium , except that with isotopic concentration exceeding 80 per cent in plutonium-238 ; uranium-233 ; uranium enriched in the isotopes 235 or 233 ; uranium containing the mixture of isotopes as occurring in nature other than in the form o
 f ore or ore residue ; or any material containing one or more of the foregoing ; Whereby \u201c uranium enriched in the isotope 235 or 233 \u201d means uranium containing the isotope 235 or 233 or both in an amount such that the abundance ratio of the sum of these isotopes to the isotope 238 is greater than the ratio of the isotope 235 to the isotope 238 occurring in nature . \u201c Nuclear facility \u201d means : ( a ) Any nuclear reactor , including reactors installed on vessels , vehicles , aircraft or space objects for use as an energy source in order to propel such vessels , vehicles , aircraft or space objects or for any other purpose ; ( b ) Any plant or conveyance being used for the production , storage , processing or transport of radioactive material . \u201c Device \u201d means : ( a ) Any nuclear explosive device ; or ( b ) Any radioactive material dispersal or radiation-emitting device which may , owing to its radiological properties , cause death , serious bodily injury or substantial d
 amage to property or the environment . \u201c State or government facility \u201d includes any permanent or temporary facility or conveyance that is used or occupied by representatives of a State , members of Government , the legislature or the judiciary or by officials or employees of a State or any other public authority or entity or by employees or officials of an intergovernmental organization in connection with their official duties . \u201c Military forces of a State \u201d means the armed forces of a State which are organized , trained and equipped under its internal law for the primary purpose of national defence or security and persons acting in support of those armed forces who are under their formal command , control and responsibility . Article 2 Any person commits an offence within the meaning of this Convention if that person unlawfully and intentionally : ( a ) Possesses radioactive material or makes or possesses a device : ( i ) With the intent to cause death or serious bodily i
 njury ; or ( ii ) With the intent to cause substantial damage to property or the environment ; ( b ) Uses in any way radioactive material or a device , or uses or damages a nuclear facility in a manner which releases or risks the release of radioactive material : ( i ) With the intent to cause death or serious bodily injury ; or ( ii ) With the intent to cause substantial damage to property or the environment ; or ( iii ) With the intent to compel a natural or legal person , an international organization or a State to do or refrain from doing an act . Any person also commits an offence if that person : ( a ) Threatens , under circumstances which indicate the credibility of the threat , to commit an offence as set forth in subparagraph 1 ( b ) of the present article ; or ( b ) Demands unlawfully and intentionally radioactive material , a device or a nuclear facility by threat , under circumstances which indicate the credibility of the threat , or by use of force . Any person also com
 mits an offence if that person attempts to commit an offence as set forth in paragraph 1 of the present article . Any person also commits an offence if that person : ( a ) Participates as an accomplice in an offence as set forth in paragraph 1 , 2 or 3 of the present article ; or ( b ) Organizes or directs others to commit an offence as set forth in paragraph 1 , 2 or 3 of the present article ; or ( c ) In any other way contributes to the commission of one or more offences as set forth in paragraph 1 , 2 or 3 of the present article by a group of persons acting with a common purpose ; such contribution shall be intentional and either be made with the aim of furthering the general criminal activity or purpose of the group or be made in the knowledge of the intention of the group to commit the offence or offences concerned . Article 3 This Convention shall not apply where the offence is committed within a single State , the alleged offender and the victims are nationals of that State ,
  the alleged offender is found in the territory of that State and no other State has a basis under article 9 , paragraph 1 or paragraph 2 , to exercise jurisdiction , except that the provisions of articles 7 , 12 , 14 , 15 , 16 and 17 shall , as appropriate , apply in those cases . Article 4 Nothing in this Convention shall affect other rights , obligations and responsibilities of States and individuals under international law , in particular the purposes and principles of the Charter of the United Nations and international humanitarian law . The activities of armed forces during an armed conflict , as those terms are understood under international humanitarian law , which are governed by that law are not governed by this Convention , and the activities undertaken by military forces of a State in the exercise of their official duties , inasmuch as they are governed by other rules of international law , are not governed by this Convention . The provisions of paragraph 2 of the presen
 t article shall not be interpreted as condoning or making lawful otherwise unlawful acts , or precluding prosecution under other laws . This Convention does not address , nor can it be interpreted as addressing , in any way , the issue of the legality of the use or threat of use of nuclear weapons by States . Article 5 Each State Party shall adopt such measures as may be necessary : ( a ) To establish as criminal offences under its national law the offences set forth in article 2 ; ( b ) To make those offences punishable by appropriate penalties which take into account the grave nature of these offences . Article 6 Each State Party shall adopt such measures as may be necessary , including , where appropriate , domestic legislation , to ensure that criminal acts within the scope of this Convention , in particular where they are intended or calculated to provoke a state of terror in the general public or in a group of persons or particular persons , are under no circumstances justifia
 ble by considerations of a political , philosophical , ideological , racial , ethnic , religious or other similar nature and are punished by penalties consistent with their grave nature . Article 7 States Parties shall cooperate by : ( a ) Taking all practicable measures , including , if necessary , adapting their national law , to prevent and counter preparations in their respective territories for the commission within or outside their territories of the offences set forth in article 2 , including measures to prohibit in their territories illegal activities of persons , groups and organizations that encourage , instigate , organize , knowingly finance or knowingly provide technical assistance or information or engage in the perpetration of those offences ; ( b ) Exchanging accurate and verified information in accordance with their national law and in the manner of and subject to the conditions specified herein , and coordinating administrative and other measures taken as appropria
 te to detect , prevent , suppress and investigate the offences set forth in article 2 and also in order to institute criminal proceedings against persons alleged to have committed those crimes . In particular , a State Party shall take appropriate measures in order to inform without delay the other States referred to in article 9 in respect of the commission of the offences set forth in article 2 as well as preparations to commit such offences about which it has learned , and also to inform , where appropriate , international organizations . States Parties shall take appropriate measures consistent with their national law to protect the confidentiality of any information which they receive in confidence by virtue of the provisions of this Convention from another State Party or through participation in an activity carried out for the implementation of this Convention . If States Parties provide information to international organizations in confidence , steps shall be taken to ensure 
 that the confidentiality of such information is protected . States Parties shall not be required by this Convention to provide any information which they are not permitted to communicate pursuant to national law or which would jeopardize the security of the State concerned or the physical protection of nuclear material . States Parties shall inform the Secretary-General of the United Nations of their competent authorities and liaison points responsible for sending and receiving the information referred to in the present article . The Secretary-General of the United Nations shall communicate such information regarding competent authorities and liaison points to all States Parties and the International Atomic Energy Agency . Such authorities and liaison points must be accessible on a continuous basis . Article 8 For purposes of preventing offences under this Convention , States Parties shall make every effort to adopt appropriate measures to ensure the protection of radioactive materi
 al , taking into account relevant recommendations and functions of the International Atomic Energy Agency . Article 9 Each State Party shall take such measures as may be necessary to establish its jurisdiction over the offences set forth in article 2 when : ( a ) The offence is committed in the territory of that State ; or ( b ) The offence is committed on board a vessel flying the flag of that State or an aircraft which is registered under the laws of that State at the time the offence is committed ; or ( c ) The offence is committed by a national of that State . A State Party may also establish its jurisdiction over any such offence when : ( a ) The offence is committed against a national of that State ; or ( b ) The offence is committed against a State or government facility of that State abroad , including an embassy or other diplomatic or consular premises of that State ; or ( c ) The offence is committed by a stateless person who has his or her habitual residence in the territ
 ory of that State ; or ( d ) The offence is committed in an attempt to compel that State to do or abstain from doing any act ; or ( e ) The offence is committed on board an aircraft which is operated by the Government of that State . Upon ratifying , accepting , approving or acceding to this Convention , each State Party shall notify the Secretary-General of the United Nations of the jurisdiction it has established under its national law in accordance with paragraph 2 of the present article . Should any change take place , the State Party concerned shall immediately notify the Secretary-General . Each State Party shall likewise take such measures as may be necessary to establish its jurisdiction over the offences set forth in article 2 in cases where the alleged offender is present in its territory and it does not extradite that person to any of the States Parties which have established their jurisdiction in accordance with paragraph 1 or 2 of the present article . This Convention d
 oes not exclude the exercise of any criminal jurisdiction established by a State Party in accordance with its national law . Article 10 Upon receiving information that an offence set forth in article 2 has been committed or is being committed in the territory of a State Party or that a person who has committed or who is alleged to have committed such an offence may be present in its territory , the State Party concerned shall take such measures as may be necessary under its national law to investigate the facts contained in the information . Upon being satisfied that the circumstances so warrant , the State Party in whose territory the offender or alleged offender is present shall take the appropriate measures under its national law so as to ensure that person 's presence for the purpose of prosecution or extradition . Any person regarding whom the measures referred to in paragraph 2 of the present article are being taken shall be entitled to : ( a ) Communicate without delay with t
 he nearest appropriate representative of the State of which that person is a national or which is otherwise entitled to protect that person 's rights or , if that person is a stateless person , the State in the territory of which that person habitually resides ; ( b ) Be visited by a representative of that State ; ( c ) Be informed of that person 's rights under subparagraphs ( a ) and ( b ) . The rights referred to in paragraph 3 of the present article shall be exercised in conformity with the laws and regulations of the State in the territory of which the offender or alleged offender is present , subject to the provision that the said laws and regulations must enable full effect to be given to the purposes for which the rights accorded under paragraph 3 are intended . The provisions of paragraphs 3 and 4 of the present article shall be without prejudice to the right of any State Party having a claim to jurisdiction in accordance with article 9 , subparagraph 1 ( c ) or 2 ( c ) , t
 o invite the International Committee of the Red Cross to communicate with and visit the alleged offender . When a State Party , pursuant to the present article , has taken a person into custody , it shall immediately notify , directly or through the Secretary-General of the United Nations , the States Parties which have established jurisdiction in accordance with article 9 , paragraphs 1 and 2 and , if it considers it advisable , any other interested States Parties , of the fact that that person is in custody and of the circumstances which warrant that person 's detention . The State which makes the investigation contemplated in paragraph 1 of the present article shall promptly inform the said States Parties of its findings and shall indicate whether it intends to exercise jurisdiction . Article 11 The State Party in the territory of which the alleged offender is present shall , in cases to which article 9 applies , if it does not extradite that person , be obliged , without excepti
 on whatsoever and whether or not the offence was committed in its territory , to submit the case without undue delay to its competent authorities for the purpose of prosecution , through proceedings in accordance with the laws of that State . Those authorities shall take their decision in the same manner as in the case of any other offence of a grave nature under the law of that State . Whenever a State Party is permitted under its national law to extradite or otherwise surrender one of its nationals only upon the condition that the person will be returned to that State to serve the sentence imposed as a result of the trial or proceeding for which the extradition or surrender of the person was sought , and this State and the State seeking the extradition of the person agree with this option and other terms they may deem appropriate , such a conditional extradition or surrender shall be sufficient to discharge the obligation set forth in paragraph 1 of the present article . Article 1
 2 Any person who is taken into custody or regarding whom any other measures are taken or proceedings are carried out pursuant to this Convention shall be guaranteed fair treatment , including enjoyment of all rights and guarantees in conformity with the law of the State in the territory of which that person is present and applicable provisions of international law , including international law of human rights . Article 13 The offences set forth in article 2 shall be deemed to be included as extraditable offences in any extradition treaty existing between any of the States Parties before the entry into force of this Convention . States Parties undertake to include such offences as extraditable offences in every extradition treaty to be subsequently concluded between them . When a State Party which makes extradition conditional on the existence of a treaty receives a request for extradition from another State Party with which it has no extradition treaty , the requested State Party ma
 y , at its option , consider this Convention as a legal basis for extradition in respect of the offences set forth in article 2. Extradition shall be subject to the other conditions provided by the law of the requested State . States Parties which do not make extradition conditional on the existence of a treaty shall recognize the offences set forth in article 2 as extraditable offences between themselves , subject to the conditions provided by the law of the requested State . If necessary , the offences set forth in article 2 shall be treated , for the purposes of extradition between States Parties , as if they had been committed not only in the place in which they occurred but also in the territory of the States that have established jurisdiction in accordance with article 9 , paragraphs 1 and 2. The provisions of all extradition treaties and arrangements between States Parties with regard to offences set forth in article 2 shall be deemed to be modified as between States Parties 
 to the extent that they are incompatible with this Convention . Article 14 States Parties shall afford one another the greatest measure of assistance in connection with investigations or criminal or extradition proceedings brought in respect of the offences set forth in article 2 , including assistance in obtaining evidence at their disposal necessary for the proceedings . States Parties shall carry out their obligations under paragraph 1 of the present article in conformity with any treaties or other arrangements on mutual legal assistance that may exist between them . In the absence of such treaties or arrangements , States Parties shall afford one another assistance in accordance with their national law . Article 15 None of the offences set forth in article 2 shall be regarded , for the purposes of extradition or mutual legal assistance , as a political offence or as an offence connected with a political offence or as an offence inspired by political motives . Accordingly , a req
 uest for extradition or for mutual legal assistance based on such an offence may not be refused on the sole ground that it concerns a political offence or an offence connected with a political offence or an offence inspired by political motives . Article 16 Nothing in this Convention shall be interpreted as imposing an obligation to extradite or to afford mutual legal assistance if the requested State Party has substantial grounds for believing that the request for extradition for offences set forth in article 2 or for mutual legal assistance with respect to such offences has been made for the purpose of prosecuting or punishing a person on account of that person 's race , religion , nationality , ethnic origin or political opinion or that compliance with the request would cause prejudice to that person 's position for any of these reasons . Article 17 A person who is being detained or is serving a sentence in the territory of one State Party whose presence in another State Party is
  requested for purposes of testimony , identification or otherwise providing assistance in obtaining evidence for the investigation or prosecution of offences under this Convention may be transferred if the following conditions are met : ( a ) The person freely gives his or her informed consent ; and ( b ) The competent authorities of both States agree , subject to such conditions as those States may deem appropriate . For the purposes of the present article : ( a ) The State to which the person is transferred shall have the authority and obligation to keep the person transferred in custody , unless otherwise requested or authorized by the State from which the person was transferred ; ( b ) The State to which the person is transferred shall without delay implement its obligation to return the person to the custody of the State from which the person was transferred as agreed beforehand , or as otherwise agreed , by the competent authorities of both States ; ( c ) The State to which t
 he person is transferred shall not require the State from which the person was transferred to initiate extradition proceedings for the return of the person ; ( d ) The person transferred shall receive credit for service of the sentence being served in the State from which he was transferred for time spent in the custody of the State to which he was transferred . Unless the State Party from which a person is to be transferred in accordance with the present article so agrees , that person , whatever his or her nationality , shall not be prosecuted or detained or subjected to any other restriction of his or her personal liberty in the territory of the State to which that person is transferred in respect of acts or convictions anterior to his or her departure from the territory of the State from which such person was transferred . Article 18 Upon seizing or otherwise taking control of radioactive material , devices or nuclear facilities , following the commission of an offence set forth
  in article 2 , the State Party in possession of it shall : ( a ) Take steps to render harmless the radioactive material , device or nuclear facility ; ( b ) Ensure that any nuclear material is held in accordance with applicable International Atomic Energy Agency safeguards ; and ( c ) Have regard to physical protection recommendations and health and safety standards published by the International Atomic Energy Agency . Upon the completion of any proceedings connected with an offence set forth in article 2 , or sooner if required by international law , any radioactive material , device or nuclear facility shall be returned , after consultations ( in particular , regarding modalities of return and storage ) with the States Parties concerned to the State Party to which it belongs , to the State Party of which the natural or legal person owning such radioactive material , device or facility is a national or resident , or to the State Party from whose territory it was stolen or otherwis
 e unlawfully obtained . 3(1 ) Where a State Party is prohibited by national or international law from returning or accepting such radioactive material , device or nuclear facility or where the States Parties concerned so agree , subject to paragraph 3(2 ) of the present article , the State Party in possession of the radioactive material , devices or nuclear facilities shall continue to take the steps described in paragraph 1 of the present article ; such radioactive material , devices or nuclear facilities shall be used only for peaceful purposes . 3(2 ) Where it is not lawful for the State Party in possession of the radioactive material , devices or nuclear facilities to possess them , that State shall ensure that they are as soon as possible placed in the possession of a State for which such possession is lawful and which , where appropriate , has provided assurances consistent with the requirements of paragraph 1 of the present article in consultation with that State , for the pu
 rpose of rendering it harmless ; such radioactive material , devices or nuclear facilities shall be used only for peaceful purposes . If the radioactive material , devices or nuclear facilities referred to in paragraphs 1 and 2 of the present article do not belong to any of the States Parties or to a national or resident of a State Party or was not stolen or otherwise unlawfully obtained from the territory of a State Party , or if no State is willing to receive such item pursuant to paragraph 3 of the present article , a separate decision concerning its disposition shall , subject to paragraph 3(2 ) of the present article , be taken after consultations between the States concerned and any relevant international organizations . For the purposes of paragraphs 1 , 2 , 3 and 4 of the present article , the State Party in possession of the radioactive material , device or nuclear facility may request the assistance and cooperation of other States Parties , in particular the States Parties
  concerned , and any relevant international organizations , in particular the International Atomic Energy Agency . States Parties and the relevant international organizations are encouraged to provide assistance pursuant to this paragraph to the maximum extent possible . The States Parties involved in the disposition or retention of the radioactive material , device or nuclear facility pursuant to the present article shall inform the Director General of the International Atomic Energy Agency of the manner in which such an item was disposed of or retained . The Director General of the International Atomic Energy Agency shall transmit the information to the other States Parties . In the event of any dissemination in connection with an offence set forth in article 2 , nothing in the present article shall affect in any way the rules of international law governing liability for nuclear damage , or other rules of international law . Article 19 The State Party where the alleged offender is
  prosecuted shall , in accordance with its national law or applicable procedures , communicate the final outcome of the proceedings to the Secretary-General of the United Nations , who shall transmit the information to the other States Parties . Article 20 States Parties shall conduct consultations with one another directly or through the Secretary-General of the United Nations , with the assistance of international organizations as necessary , to ensure effective implementation of this Convention . Article 21 The States Parties shall carry out their obligations under this Convention in a manner consistent with the principles of sovereign equality and territorial integrity of States and that of non-intervention in the domestic affairs of other States . Article 22 Nothing in this Convention entitles a State Party to undertake in the territory of another State Party the exercise of jurisdiction and performance of functions which are exclusively reserved for the authorities of that oth
 er State Party by its national law . Article 23 Any dispute between two or more States Parties concerning the interpretation or application of this Convention which cannot be settled through negotiation within a reasonable time shall , at the request of one of them , be submitted to arbitration . If , within six months from the date of the request for arbitration , the parties are unable to agree on the organization of the arbitration , any one of those parties may refer the dispute to the International Court of Justice , by application , in conformity with the Statute of the Court . Each State may , at the time of signature , ratification , acceptance or approval of this Convention or accession thereto , declare that it does not consider itself bound by paragraph 1 of the present article . The other States Parties shall not be bound by paragraph 1 with respect to any State Party which has made such a reservation . Any State which has made a reservation in accordance with paragraph 
 2 of the present article may at any time withdraw that reservation by notification to the Secretary-General of the United Nations . Article 24 This Convention shall be open for signature by all States from ____________ until ____________ at United Nations Headquarters in New York . This Convention is subject to ratification , acceptance or approval . The instruments of ratification , acceptance or approval shall be deposited with the Secretary-General of the United Nations . This Convention shall be open to accession by any State . The instruments of accession shall be deposited with the Secretary-General of the United Nations . Article 25 This Convention shall enter into force on the thirtieth day following the date of the deposit of the twenty-second instrument of ratification , acceptance , approval or accession with the Secretary-General of the United Nations . For each State ratifying , accepting , approving or acceding to the Convention after the deposit of the twenty-second i
 nstrument of ratification , acceptance , approval or accession , the Convention shall enter into force on the thirtieth day after deposit by such State of its instrument of ratification , acceptance , approval or accession . Article 26 A State Party may propose an amendment to this Convention . The proposed amendment shall be submitted to the Depositary , who circulates it immediately to all States Parties . If the majority of the States Parties request the Depositary to convene a Conference to consider the proposed amendments , the Depositary shall invite all States Parties to attend such a Conference to begin not sooner than three months after the invitations are issued . The Conference shall make every effort to ensure amendments are adopted by consensus . Should this not be possible , amendments shall be adopted by a two-thirds majority of all States Parties . Any amendment adopted at the Conference shall be promptly circulated by the Depositary to all States Parties . The amend
 ment adopted pursuant to paragraph 3 of the present article shall enter into force for each State Party that deposits its instrument of ratification , acceptance , accession or approval of the amendment on the thirtieth day after the date on which two thirds of the States Parties have deposited their relevant instrument . Thereafter , the amendment shall enter into force for any State Party on the thirtieth day after the date on which that State deposits its relevant instrument . Article 27 Any State Party may denounce this Convention by written notification to the Secretary-General of the United Nations . Denunciation shall take effect one year following the date on which notification is received by the Secretary-General of the United Nations . Article 28 The original of this Convention , of which the Arabic , Chinese , English , French , Russian and Spanish texts are equally authentic , shall be deposited with the Secretary-General of the United Nations , who shall send certified 
 copies thereof to all States . In witness whereof , the undersigned , being duly authorized thereto by their respective Governments , have signed this Convention , opened for signature at United Nations Headquarters in New York on __________________ . General Assembly resolution 50/6 of 24 October 1995. 
\ No newline at end of file


[35/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/52FictDostoyevskyF_CrimePun_ep_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/52FictDostoyevskyF_CrimePun_ep_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/52FictDostoyevskyF_CrimePun_ep_EN.txt.txt
new file mode 100644
index 0000000..38013c8
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/52FictDostoyevskyF_CrimePun_ep_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Epilogue I Siberia . On the banks of a broad solitary river stands a town , one of the administrative centres of Russia ; in the town there is a fortress , in the fortress there is a prison . In the prison the second-class convict Rodion Raskolnikov has been confined for nine months . Almost a year and a half has passed since his crime . There had been little difficulty about his trial . The criminal adhered exactly , firmly , and clearly to his statement . He did not confuse nor misrepresent the facts , nor soften them in his own interest , nor omit the smallest detail . He explained every incident of the murder , the secret of the pledge ( the piece of wood with a strip of metal ) which was found in the murdered woman 's hand . He described minutely how he had taken her keys , what they were like , as well as the chest and its contents ; he explained the mystery of Lizaveta 's murder ; described how Koch and , after him , the student knocked , and repeated all they had said to
  one another ; how he afterwards had run downstairs and heard Nikolay and Dmitri shouting ; how he had hidden in the empty flat and afterwards gone home . He ended by indicating the stone in the yard off the Voznesensky Prospect under which the purse and the trinkets were found . The whole thing , in fact , was perfectly clear . The lawyers and the judges were very much struck , among other things , by the fact that he had hidden the trinkets and the purse under a stone , without making use of them , and that , what was more , he did not now remember what the trinkets were like , or even how many there were . The fact that he had never opened the purse and did not even know how much was in it seemed incredible . There turned out to be in the purse three hundred and seventeen roubles and sixty copecks . From being so long under the stone , some of the most valuable notes lying uppermost had suffered from the damp . They were a long while trying to discover why the accused man should 
 tell a lie about this , when about everything else he had made a truthful and straightforward confession . Finally some of the lawyers more versed in psychology admitted that it was possible he had really not looked into the purse , and so did n't know what was in it when he hid it under the stone . But they immediately drew the deduction that the crime could only have been committed through temporary mental derangement , through homicidal mania , without object or the pursuit of gain . This fell in with the most recent fashionable theory of temporary insanity , so often applied in our days in criminal cases . Moreover Raskolnikov 's hypochondriacal condition was proved by many witnesses , by Dr. Zossimov , his former fellow students , his landlady and her servant . All this pointed strongly to the conclusion that Raskolnikov was not quite like an ordinary murderer and robber , but that there was another element in the case . To the intense annoyance of those who maintained this opi
 nion , the criminal scarcely attempted to defend himself . To the decisive question as to what motive impelled him to the murder and the robbery , he answered very clearly with the coarsest frankness that the cause was his miserable position , his poverty and helplessness , and his desire to provide for his first steps in life by the help of the three thousand roubles he had reckoned on finding . He had been led to the murder through his shallow and cowardly nature , exasperated moreover by privation and failure . To the question what led him to confess , he answered that it was his heartfelt repentance . All this was almost coarse . . . . The sentence however was more merciful than could have been expected , perhaps partly because the criminal had not tried to justify himself , but had rather shown a desire to exaggerate his guilt . All the strange and peculiar circumstances of the crime were taken into consideration . There could be no doubt of the abnormal and poverty-stricken co
 ndition of the criminal at the time . The fact that he had made no use of what he had stolen was put down partly to the effect of remorse , partly to his abnormal mental condition at the time of the crime . Incidentally the murder of Lizaveta served indeed to confirm the last hypothesis : a man commits two murders and forgets that the door is open ! Finally , the confession , at the very moment when the case was hopelessly muddled by the false evidence given by Nikolay through melancholy and fanaticism , and when , moreover , there were no proofs against the real criminal , no suspicions even ( Porfiry Petrovitch fully kept his word ) --all this did much to soften the sentence . Other circumstances , too , in the prisoner 's favour came out quite unexpectedly . Razumihin somehow discovered and proved that while Raskolnikov was at the university he had helped a poor consumptive fellow student and had spent his last penny on supporting him for six months , and when this student died ,
  leaving a decrepit old father whom he had maintained almost from his thirteenth year , Raskolnikov had got the old man into a hospital and paid for his funeral when he died . Raskolnikov 's landlady bore witness , too , that when they had lived in another house at Five Corners , Raskolnikov had rescued two little children from a house on fire and was burnt in doing so . This was investigated and fairly well confirmed by many witnesses . These facts made an impression in his favour . And in the end the criminal was , in consideration of extenuating circumstances , condemned to penal servitude in the second class for a term of eight years only . At the very beginning of the trial Raskolnikov 's mother fell ill . Dounia and Razumihin found it possible to get her out of Petersburg during the trial . Razumihin chose a town on the railway not far from Petersburg , so as to be able to follow every step of the trial and at the same time to see Avdotya Romanovna as often as possible . Pulch
 eria Alexandrovna 's illness was a strange nervous one and was accompanied by a partial derangement of her intellect . When Dounia returned from her last interview with her brother , she had found her mother already ill , in feverish delirium . That evening Razumihin and she agreed what answers they must make to her mother 's questions about Raskolnikov and made up a complete story for her mother 's benefit of his having to go away to a distant part of Russia on a business commission , which would bring him in the end money and reputation . But they were struck by the fact that Pulcheria Alexandrovna never asked them anything on the subject , neither then nor thereafter . On the contrary , she had her own version of her son 's sudden departure ; she told them with tears how he had come to say good-bye to her , hinting that she alone knew many mysterious and important facts , and that Rodya had many very powerful enemies , so that it was necessary for him to be in hiding . As for his
  future career , she had no doubt that it would be brilliant when certain sinister influences could be removed . She assured Razumihin that her son would be one day a great statesman , that his article and brilliant literary talent proved it . This article she was continually reading , she even read it aloud , almost took it to bed with her , but scarcely asked where Rodya was , though the subject was obviously avoided by the others , which might have been enough to awaken her suspicions . They began to be frightened at last at Pulcheria Alexandrovna 's strange silence on certain subjects . She did not , for instance , complain of getting no letters from him , though in previous years she had only lived on the hope of letters from her beloved Rodya . This was the cause of great uneasiness to Dounia ; the idea occurred to her that her mother suspected that there was something terrible in her son 's fate and was afraid to ask , for fear of hearing something still more awful . In any c
 ase , Dounia saw clearly that her mother was not in full possession of her faculties . It happened once or twice , however , that Pulcheria Alexandrovna gave such a turn to the conversation that it was impossible to answer her without mentioning where Rodya was , and on receiving unsatisfactory and suspicious answers she became at once gloomy and silent , and this mood lasted for a long time . Dounia saw at last that it was hard to deceive her and came to the conclusion that it was better to be absolutely silent on certain points ; but it became more and more evident that the poor mother suspected something terrible . Dounia remembered her brother 's telling her that her mother had overheard her talking in her sleep on the night after her interview with Svidriga�lov and before the fatal day of the confession : had not she made out something from that ? Sometimes days and even weeks of gloomy silence and tears would be succeeded by a period of hysterical animation , and the invalid 
 would begin to talk almost incessantly of her son , of her hopes of his future . . . . Her fancies were sometimes very strange . They humoured her , pretended to agree with her ( she saw perhaps that they were pretending ) , but she still went on talking . Five months after Raskolnikov 's confession , he was sentenced . Razumihin and Sonia saw him in prison as often as it was possible . At last the moment of separation came . Dounia swore to her brother that the separation should not be for ever , Razumihin did the same . Razumihin , in his youthful ardour , had firmly resolved to lay the foundations at least of a secure livelihood during the next three or four years , and saving up a certain sum , to emigrate to Siberia , a country rich in every natural resource and in need of workers , active men and capital . There they would settle in the town where Rodya was and all together would begin a new life . They all wept at parting . Raskolnikov had been very dreamy for a few days befo
 re . He asked a great deal about his mother and was constantly anxious about her . He worried so much about her that it alarmed Dounia . When he heard about his mother 's illness he became very gloomy . With Sonia he was particularly reserved all the time . With the help of the money left to her by Svidriga�lov , Sonia had long ago made her preparations to follow the party of convicts in which he was despatched to Siberia . Not a word passed between Raskolnikov and her on the subject , but both knew it would be so . At the final leave-taking he smiled strangely at his sister 's and Razumihin 's fervent anticipations of their happy future together when he should come out of prison . He predicted that their mother 's illness would soon have a fatal ending . Sonia and he at last set off . Two months later Dounia was married to Razumihin . It was a quiet and sorrowful wedding ; Porfiry Petrovitch and Zossimov were invited however . During all this period Razumihin wore an air of resolu
 te determination . Dounia put implicit faith in his carrying out his plans and indeed she could not but believe in him . He displayed a rare strength of will . Among other things he began attending university lectures again in order to take his degree . They were continually making plans for the future ; both counted on settling in Siberia within five years at least . Till then they rested their hopes on Sonia . Pulcheria Alexandrovna was delighted to give her blessing to Dounia 's marriage with Razumihin ; but after the marriage she became even more melancholy and anxious . To give her pleasure Razumihin told her how Raskolnikov had looked after the poor student and his decrepit father and how a year ago he had been burnt and injured in rescuing two little children from a fire . These two pieces of news excited Pulcheria Alexandrovna 's disordered imagination almost to ecstasy . She was continually talking about them , even entering into conversation with strangers in the street , 
 though Dounia always accompanied her . In public conveyances and shops , wherever she could capture a listener , she would begin the discourse about her son , his article , how he had helped the student , how he had been burnt at the fire , and so on ! Dounia did not know how to restrain her . Apart from the danger of her morbid excitement , there was the risk of someone 's recalling Raskolnikov 's name and speaking of the recent trial . Pulcheria Alexandrovna found out the address of the mother of the two children her son had saved and insisted on going to see her . At last her restlessness reached an extreme point . She would sometimes begin to cry suddenly and was often ill and feverishly delirious . One morning she declared that by her reckoning Rodya ought soon to be home , that she remembered when he said good-bye to her he said that they must expect him back in nine months . She began to prepare for his coming , began to do up her room for him , to clean the furniture , to wa
 sh and put up new hangings and so on . Dounia was anxious , but said nothing and helped her to arrange the room . After a fatiguing day spent in continual fancies , in joyful day-dreams and tears , Pulcheria Alexandrovna was taken ill in the night and by morning she was feverish and delirious . It was brain fever . She died within a fortnight . In her delirium she dropped words which showed that she knew a great deal more about her son 's terrible fate than they had supposed . For a long time Raskolnikov did not know of his mother 's death , though a regular correspondence had been maintained from the time he reached Siberia . It was carried on by means of Sonia , who wrote every month to the Razumihins and received an answer with unfailing regularity . At first they found Sonia 's letters dry and unsatisfactory , but later on they came to the conclusion that the letters could not be better , for from these letters they received a complete picture of their unfortunate brother 's lif
 e . Sonia 's letters were full of the most matter-of-fact detail , the simplest and clearest description of all Raskolnikov 's surroundings as a convict . There was no word of her own hopes , no conjecture as to the future , no description of her feelings . Instead of any attempt to interpret his state of mind and inner life , she gave the simple facts--that is , his own words , an exact account of his health , what he asked for at their interviews , what commission he gave her and so on . All these facts she gave with extraordinary minuteness . The picture of their unhappy brother stood out at last with great clearness and precision . There could be no mistake , because nothing was given but facts . But Dounia and her husband could get little comfort out of the news , especially at first . Sonia wrote that he was constantly sullen and not ready to talk , that he scarcely seemed interested in the news she gave him from their letters , that he sometimes asked after his mother and tha
 t when , seeing that he had guessed the truth , she told him at last of her death , she was surprised to find that he did not seem greatly affected by it , not externally at any rate . She told them that , although he seemed so wrapped up in himself and , as it were , shut himself off from everyone--he took a very direct and simple view of his new life ; that he understood his position , expected nothing better for the time , had no ill-founded hopes ( as is so common in his position ) and scarcely seemed surprised at anything in his surroundings , so unlike anything he had known before . She wrote that his health was satisfactory ; he did his work without shirking or seeking to do more ; he was almost indifferent about food , but except on Sundays and holidays the food was so bad that at last he had been glad to accept some money from her , Sonia , to have his own tea every day . He begged her not to trouble about anything else , declaring that all this fuss about him only annoyed 
 him . Sonia wrote further that in prison he shared the same room with the rest , that she had not seen the inside of their barracks , but concluded that they were crowded , miserable and unhealthy ; that he slept on a plank bed with a rug under him and was unwilling to make any other arrangement . But that he lived so poorly and roughly , not from any plan or design , but simply from inattention and indifference . Sonia wrote simply that he had at first shown no interest in her visits , had almost been vexed with her indeed for coming , unwilling to talk and rude to her . But that in the end these visits had become a habit and almost a necessity for him , so that he was positively distressed when she was ill for some days and could not visit him . She used to see him on holidays at the prison gates or in the guard-room , to which he was brought for a few minutes to see her . On working days she would go to see him at work either at the workshops or at the brick kilns , or at the she
 ds on the banks of the Irtish . About herself , Sonia wrote that she had succeeded in making some acquaintances in the town , that she did sewing , and , as there was scarcely a dressmaker in the town , she was looked upon as an indispensable person in many houses . But she did not mention that the authorities were , through her , interested in Raskolnikov ; that his task was lightened and so on . At last the news came ( Dounia had indeed noticed signs of alarm and uneasiness in the preceding letters ) that he held aloof from everyone , that his fellow prisoners did not like him , that he kept silent for days at a time and was becoming very pale . In the last letter Sonia wrote that he had been taken very seriously ill and was in the convict ward of the hospital . II He was ill a long time . But it was not the horrors of prison life , not the hard labour , the bad food , the shaven head , or the patched clothes that crushed him . What did he care for all those trials and hardships !
  he was even glad of the hard work . Physically exhausted , he could at least reckon on a few hours of quiet sleep . And what was the food to him--the thin cabbage soup with beetles floating in it ? In the past as a student he had often not had even that . His clothes were warm and suited to his manner of life . He did not even feel the fetters . Was he ashamed of his shaven head and parti-coloured coat ? Before whom ? Before Sonia ? Sonia was afraid of him , how could he be ashamed before her ? And yet he was ashamed even before Sonia , whom he tortured because of it with his contemptuous rough manner . But it was not his shaven head and his fetters he was ashamed of : his pride had been stung to the quick . It was wounded pride that made him ill . Oh , how happy he would have been if he could have blamed himself ! He could have borne anything then , even shame and disgrace . But he judged himself severely , and his exasperated conscience found no particularly terrible fault in his
  past , except a simple blunder which might happen to anyone . He was ashamed just because he , Raskolnikov , had so hopelessly , stupidly come to grief through some decree of blind fate , and must humble himself and submit to " the idiocy " of a sentence , if he were anyhow to be at peace . Vague and objectless anxiety in the present , and in the future a continual sacrifice leading to nothing--that was all that lay before him . And what comfort was it to him that at the end of eight years he would only be thirty-two and able to begin a new life ! What had he to live for ? What had he to look forward to ? Why should he strive ? To live in order to exist ? Why , he had been ready a thousand times before to give up existence for the sake of an idea , for a hope , even for a fancy . Mere existence had always been too little for him ; he had always wanted more . Perhaps it was just because of the strength of his desires that he had thought himself a man to whom more was permissible tha
 n to others . And if only fate would have sent him repentance--burning repentance that would have torn his heart and robbed him of sleep , that repentance , the awful agony of which brings visions of hanging or drowning ! Oh , he would have been glad of it ! Tears and agonies would at least have been life . But he did not repent of his crime . At least he might have found relief in raging at his stupidity , as he had raged at the grotesque blunders that had brought him to prison . But now in prison , in freedom , he thought over and criticised all his actions again and by no means found them so blundering and so grotesque as they had seemed at the fatal time . " In what way , " he asked himself , " was my theory stupider than others that have swarmed and clashed from the beginning of the world ? One has only to look at the thing quite independently , broadly , and uninfluenced by commonplace ideas , and my idea will by no means seem so . . . strange . Oh , sceptics and halfpenny phi
 losophers , why do you halt half-way ! " " Why does my action strike them as so horrible ? " he said to himself . " Is it because it was a crime ? What is meant by crime ? My conscience is at rest . Of course , it was a legal crime , of course , the letter of the law was broken and blood was shed . Well , punish me for the letter of the law . . . and that 's enough . Of course , in that case many of the benefactors of mankind who snatched power for themselves instead of inheriting it ought to have been punished at their first steps . But those men succeeded and so they were right , and I did n't , and so I had no right to have taken that step . " It was only in that that he recognised his criminality , only in the fact that he had been unsuccessful and had confessed it . He suffered too from the question : why had he not killed himself ? Why had he stood looking at the river and preferred to confess ? Was the desire to live so strong and was it so hard to overcome it ? Had not Svidr
 iga�lov overcome it , although he was afraid of death ? In misery he asked himself this question , and could not understand that , at the very time he had been standing looking into the river , he had perhaps been dimly conscious of the fundamental falsity in himself and his convictions . He did n't understand that that consciousness might be the promise of a future crisis , of a new view of life and of his future resurrection . He preferred to attribute it to the dead weight of instinct which he could not step over , again through weakness and meanness . He looked at his fellow prisoners and was amazed to see how they all loved life and prized it . It seemed to him that they loved and valued life more in prison than in freedom . What terrible agonies and privations some of them , the tramps for instance , had endured ! Could they care so much for a ray of sunshine , for the primeval forest , the cold spring hidden away in some unseen spot , which the tramp had marked three years b
 efore , and longed to see again , as he might to see his sweetheart , dreaming of the green grass round it and the bird singing in the bush ? As he went on he saw still more inexplicable examples . In prison , of course , there was a great deal he did not see and did not want to see ; he lived as it were with downcast eyes . It was loathsome and unbearable for him to look . But in the end there was much that surprised him and he began , as it were involuntarily , to notice much that he had not suspected before . What surprised him most of all was the terrible impossible gulf that lay between him and all the rest . They seemed to be a different species , and he looked at them and they at him with distrust and hostility . He felt and knew the reasons of his isolation , but he would never have admitted till then that those reasons were so deep and strong . There were some Polish exiles , political prisoners , among them . They simply looked down upon all the rest as ignorant churls ; b
 ut Raskolnikov could not look upon them like that . He saw that these ignorant men were in many respects far wiser than the Poles . There were some Russians who were just as contemptuous , a former officer and two seminarists . Raskolnikov saw their mistake as clearly . He was disliked and avoided by everyone ; they even began to hate him at last--why , he could not tell . Men who had been far more guilty despised and laughed at his crime . " You 're a gentleman , " they used to say . " You should n't hack about with an axe ; that 's not a gentleman 's work . " The second week in Lent , his turn came to take the sacrament with his gang . He went to church and prayed with the others . A quarrel broke out one day , he did not know how . All fell on him at once in a fury . " You 're an infidel ! You do n't believe in God , " they shouted . " You ought to be killed . " He had never talked to them about God nor his belief , but they wanted to kill him as an infidel . He said nothing . On
 e of the prisoners rushed at him in a perfect frenzy . Raskolnikov awaited him calmly and silently ; his eyebrows did not quiver , his face did not flinch . The guard succeeded in intervening between him and his assailant , or there would have been bloodshed . There was another question he could not decide : why were they all so fond of Sonia ? She did not try to win their favour ; she rarely met them , sometimes only she came to see him at work for a moment . And yet everybody knew her , they knew that she had come out to follow him , knew how and where she lived . She never gave them money , did them no particular services . Only once at Christmas she sent them all presents of pies and rolls . But by degrees closer relations sprang up between them and Sonia . She would write and post letters for them to their relations . Relations of the prisoners who visited the town , at their instructions , left with Sonia presents and money for them . Their wives and sweethearts knew her and u
 sed to visit her . And when she visited Raskolnikov at work , or met a party of the prisoners on the road , they all took off their hats to her . " Little mother Sofya Semyonovna , you are our dear , good little mother , " coarse branded criminals said to that frail little creature . She would smile and bow to them and everyone was delighted when she smiled . They even admired her gait and turned round to watch her walking ; they admired her too for being so little , and , in fact , did not know what to admire her most for . They even came to her for help in their illnesses . He was in the hospital from the middle of Lent till after Easter . When he was better , he remembered the dreams he had had while he was feverish and delirious . He dreamt that the whole world was condemned to a terrible new strange plague that had come to Europe from the depths of Asia . All were to be destroyed except a very few chosen . Some new sorts of microbes were attacking the bodies of men , but these 
 microbes were endowed with intelligence and will . Men attacked by them became at once mad and furious . But never had men considered themselves so intellectual and so completely in possession of the truth as these sufferers , never had they considered their decisions , their scientific conclusions , their moral convictions so infallible . Whole villages , whole towns and peoples went mad from the infection . All were excited and did not understand one another . Each thought that he alone had the truth and was wretched looking at the others , beat himself on the breast , wept , and wrung his hands . They did not know how to judge and could not agree what to consider evil and what good ; they did not know whom to blame , whom to justify . Men killed each other in a sort of senseless spite . They gathered together in armies against one another , but even on the march the armies would begin attacking each other , the ranks would be broken and the soldiers would fall on each other , sta
 bbing and cutting , biting and devouring each other . The alarm bell was ringing all day long in the towns ; men rushed together , but why they were summoned and who was summoning them no one knew . The most ordinary trades were abandoned , because everyone proposed his own ideas , his own improvements , and they could not agree . The land too was abandoned . Men met in groups , agreed on something , swore to keep together , but at once began on something quite different from what they had proposed . They accused one another , fought and killed each other . There were conflagrations and famine . All men and all things were involved in destruction . The plague spread and moved further and further . Only a few men could be saved in the whole world . They were a pure chosen people , destined to found a new race and a new life , to renew and purify the earth , but no one had seen these men , no one had heard their words and their voices . Raskolnikov was worried that this senseless drea
 m haunted his memory so miserably , the impression of this feverish delirium persisted so long . The second week after Easter had come . There were warm bright spring days ; in the prison ward the grating windows under which the sentinel paced were opened . Sonia had only been able to visit him twice during his illness ; each time she had to obtain permission , and it was difficult . But she often used to come to the hospital yard , especially in the evening , sometimes only to stand a minute and look up at the windows of the ward . One evening , when he was almost well again , Raskolnikov fell asleep . On waking up he chanced to go to the window , and at once saw Sonia in the distance at the hospital gate . She seemed to be waiting for someone . Something stabbed him to the heart at that minute . He shuddered and moved away from the window . Next day Sonia did not come , nor the day after ; he noticed that he was expecting her uneasily . At last he was discharged . On reaching the 
 prison he learnt from the convicts that Sofya Semyonovna was lying ill at home and was unable to go out . He was very uneasy and sent to inquire after her ; he soon learnt that her illness was not dangerous . Hearing that he was anxious about her , Sonia sent him a pencilled note , telling him that she was much better , that she had a slight cold and that she would soon , very soon come and see him at his work . His heart throbbed painfully as he read it . Again it was a warm bright day . Early in the morning , at six o'clock , he went off to work on the river bank , where they used to pound alabaster and where there was a kiln for baking it in a shed . There were only three of them sent . One of the convicts went with the guard to the fortress to fetch a tool ; the other began getting the wood ready and laying it in the kiln . Raskolnikov came out of the shed on to the river bank , sat down on a heap of logs by the shed and began gazing at the wide deserted river . From the high ba
 nk a broad landscape opened before him , the sound of singing floated faintly audible from the other bank . In the vast steppe , bathed in sunshine , he could just see , like black specks , the nomads ' tents . There there was freedom , there other men were living , utterly unlike those here ; there time itself seemed to stand still , as though the age of Abraham and his flocks had not passed . Raskolnikov sat gazing , his thoughts passed into day-dreams , into contemplation ; he thought of nothing , but a vague restlessness excited and troubled him . Suddenly he found Sonia beside him ; she had come up noiselessly and sat down at his side . It was still quite early ; the morning chill was still keen . She wore her poor old burnous and the green shawl ; her face still showed signs of illness , it was thinner and paler . She gave him a joyful smile of welcome , but held out her hand with her usual timidity . She was always timid of holding out her hand to him and sometimes did not of
 fer it at all , as though afraid he would repel it . He always took her hand as though with repugnance , always seemed vexed to meet her and was sometimes obstinately silent throughout her visit . Sometimes she trembled before him and went away deeply grieved . But now their hands did not part . He stole a rapid glance at her and dropped his eyes on the ground without speaking . They were alone , no one had seen them . The guard had turned away for the time . How it happened he did not know . But all at once something seemed to seize him and fling him at her feet . He wept and threw his arms round her knees . For the first instant she was terribly frightened and she turned pale . She jumped up and looked at him trembling . But at the same moment she understood , and a light of infinite happiness came into her eyes . She knew and had no doubt that he loved her beyond everything and that at last the moment had come . . . . They wanted to speak , but could not ; tears stood in their ey
 es . They were both pale and thin ; but those sick pale faces were bright with the dawn of a new future , of a full resurrection into a new life . They were renewed by love ; the heart of each held infinite sources of life for the heart of the other . They resolved to wait and be patient . They had another seven years to wait , and what terrible suffering and what infinite happiness before them ! But he had risen again and he knew it and felt it in all his being , while she--she only lived in his life . On the evening of the same day , when the barracks were locked , Raskolnikov lay on his plank bed and thought of her . He had even fancied that day that all the convicts who had been his enemies looked at him differently ; he had even entered into talk with them and they answered him in a friendly way . He remembered that now , and thought it was bound to be so . Was n't everything now bound to be changed ? He thought of her . He remembered how continually he had tormented her and wo
 unded her heart . He remembered her pale and thin little face . But these recollections scarcely troubled him now ; he knew with what infinite love he would now repay all her sufferings . And what were all , all the agonies of the past ! Everything , even his crime , his sentence and imprisonment , seemed to him now in the first rush of feeling an external , strange fact with which he had no concern . But he could not think for long together of anything that evening , and he could not have analysed anything consciously ; he was simply feeling . Life had stepped into the place of theory and something quite different would work itself out in his mind . Under his pillow lay the New Testament . He took it up mechanically . The book belonged to Sonia ; it was the one from which she had read the raising of Lazarus to him . At first he was afraid that she would worry him about religion , would talk about the gospel and pester him with books . But to his great surprise she had not once appr
 oached the subject and had not even offered him the Testament . He had asked her for it himself not long before his illness and she brought him the book without a word . Till now he had not opened it . He did not open it now , but one thought passed through his mind : " Can her convictions not be mine now ? Her feelings , her aspirations at least . . . . " She too had been greatly agitated that day , and at night she was taken ill again . But she was so happy--and so unexpectedly happy--that she was almost frightened of her happiness . Seven years , only seven years ! At the beginning of their happiness at some moments they were both ready to look on those seven years as though they were seven days. He did not know that the new life would not be given him for nothing , that he would have to pay dearly for it , that it would cost him great striving , great suffering . But that is the beginning of a new story--the story of the gradual renewal of a man , the story of his gradual regene
 ration , of his passing from one world into another , of his initiation into a new unknown life . That might be the subject of a new story , but our present story is ended . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/53FictDostoyevskyF_CrimePun_II2_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/53FictDostoyevskyF_CrimePun_II2_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/53FictDostoyevskyF_CrimePun_II2_EN.txt.txt
new file mode 100644
index 0000000..0459916
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/53FictDostoyevskyF_CrimePun_II2_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Part II , Chapter II " And what if there has been a search already ? What if I find them in my room ? " But here was his room . Nothing and no one in it . No one had peeped in . Even Nastasya had not touched it . But heavens ! how could he have left all those things in the hole ? He rushed to the corner , slipped his hand under the paper , pulled the things out and lined his pockets with them . There were eight articles in all : two little boxes with ear-rings or something of the sort , he hardly looked to see ; then four small leather cases . There was a chain , too , merely wrapped in newspaper and something else in newspaper , that looked like a decoration . . . . He put them all in the different pockets of his overcoat , and the remaining pocket of his trousers , trying to conceal them as much as possible . He took the purse , too . Then he went out of his room , leaving the door open . He walked quickly and resolutely , and though he felt shattered , he had his senses about
  him . He was afraid of pursuit , he was afraid that in another half-hour , another quarter of an hour perhaps , instructions would be issued for his pursuit , and so at all costs , he must hide all traces before then . He must clear everything up while he still had some strength , some reasoning power left him . . . . Where was he to go ? That had long been settled : " Fling them into the canal , and all traces hidden in the water , the thing would be at an end . " So he had decided in the night of his delirium when several times he had had the impulse to get up and go away , to make haste , and get rid of it all . But to get rid of it , turned out to be a very difficult task . He wandered along the bank of the Ekaterininsky Canal for half an hour or more and looked several times at the steps running down to the water , but he could not think of carrying out his plan ; either rafts stood at the steps ' edge , and women were washing clothes on them , or boats were moored there , and
  people were swarming everywhere . Moreover he could be seen and noticed from the banks on all sides ; it would look suspicious for a man to go down on purpose , stop , and throw something into the water . And what if the boxes were to float instead of sinking ? And of course they would . Even as it was , everyone he met seemed to stare and look round , as if they had nothing to do but to watch him . " Why is it , or can it be my fancy ? " he thought . At last the thought struck him that it might be better to go to the Neva . There were not so many people there , he would be less observed , and it would be more convenient in every way , above all it was further off . He wondered how he could have been wandering for a good half- hour , worried and anxious in this dangerous past without thinking of it before . And that half-hour he had lost over an irrational plan , simply because he had thought of it in delirium ! He had become extremely absent and forgetful and he was aware of it . 
 He certainly must make haste . He walked towards the Neva along V \u2014 \u2014 Prospect , but on the way another idea struck him . " Why to the Neva ? Would it not be better to go somewhere far off , to the Islands again , and there hide the things in some solitary place , in a wood or under a bush , and mark the spot perhaps ? " And though he felt incapable of clear judgment , the idea seemed to him a sound one . But he was not destined to go there . For coming out of V \u2014 \u2014 Prospect towards the square , he saw on the left a passage leading between two blank walls to a courtyard . On the right hand , the blank unwhitewashed wall of a four-storied house stretched far into the court ; on the left , a wooden hoarding ran parallel with it for twenty paces into the court , and then turned sharply to the left . Here was a deserted fenced-off place where rubbish of different sorts was lying . At the end of the court , the corner of a low , smutty , stone shed , apparently part of some works
 hop , peeped from behind the hoarding . It was probably a carriage builder 's or carpenter 's shed ; the whole place from the entrance was black with coal dust . Here would be the place to throw it , he thought . Not seeing anyone in the yard , he slipped in , and at once saw near the gate a sink , such as is often put in yards where there are many workmen or cab-drivers ; and on the hoarding above had been scribbled in chalk the time-honoured witticism , " Standing here strictly forbidden . " This was all the better , for there would be nothing suspicious about his going in . " Here I could throw it all in a heap and get away ! " Looking round once more , with his hand already in his pocket , he noticed against the outer wall , between the entrance and the sink , a big unhewn stone , weighing perhaps sixty pounds . The other side of the wall was a street . He could hear passers-by , always numerous in that part , but he could not be seen from the entrance , unless someone came in f
 rom the street , which might well happen indeed , so there was need of haste . He bent down over the stone , seized the top of it firmly in both hands , and using all his strength turned it over . Under the stone was a small hollow in the ground , and he immediately emptied his pocket into it . The purse lay at the top , and yet the hollow was not filled up . Then he seized the stone again and with one twist turned it back , so that it was in the same position again , though it stood a very little higher . But he scraped the earth about it and pressed it at the edges with his foot . Nothing could be noticed . Then he went out , and turned into the square . Again an intense , almost unbearable joy overwhelmed him for an instant , as it had in the police-office . " I have buried my tracks ! And who , who can think of looking under that stone ? It has been lying there most likely ever since the house was built , and will lie as many years more . And if it were found , who would think o
 f me ? It is all over ! No clue ! " And he laughed . Yes , he remembered that he began laughing a thin , nervous noiseless laugh , and went on laughing all the time he was crossing the square . But when he reached the K \u2014 \u2014 Boulevard where two days before he had come upon that girl , his laughter suddenly ceased . Other ideas crept into his mind . He felt all at once that it would be loathsome to pass that seat on which after the girl was gone , he had sat and pondered , and that it would be hateful , too , to meet that whiskered policeman to whom he had given the twenty copecks : " Damn him ! " He walked , looking about him angrily and distractedly . All his ideas now seemed to be circling round some single point , and he felt that there really was such a point , and that now , now , he was left facing that point\u2014and for the first time , indeed , during the last two months . " Damn it all ! " he thought suddenly , in a fit of ungovernable fury . " If it has begun , then it ha
 s begun . Hang the new life ! Good Lord , how stupid it is ! . . . And what lies I told to-day ! How despicably I fawned upon that wretched Ilya Petrovitch ! But that is all folly ! What do I care for them all , and my fawning upon them ! It is not that at all ! It is not that at all ! " Suddenly he stopped ; a new utterly unexpected and exceedingly simple question perplexed and bitterly confounded him . " If it all has really been done deliberately and not idiotically , if I really had a certain and definite object , how is it I did not even glance into the purse and do n't know what I had there , for which I have undergone these agonies , and have deliberately undertaken this base , filthy degrading business ? And here I wanted at once to throw into the water the purse together with all the things which I had not seen either . . . how 's that ? " Yes , that was so , that was all so . Yet he had known it all before , and it was not a new question for him , even when it was decided 
 in the night without hesitation and consideration , as though so it must be , as though it could not possibly be otherwise . . . . Yes , he had known it all , and understood it all ; it surely had all been settled even yesterday at the moment when he was bending over the box and pulling the jewel-cases out of it . . . . Yes , so it was . " It is because I am very ill , " he decided grimly at last , " I have been worrying and fretting myself , and I do n't know what I am doing . . . . Yesterday and the day before yesterday and all this time I have been worrying myself . . . . I shall get well and I shall not worry . . . . But what if I do n't get well at all ? Good God , how sick I am of it all ! " He walked on without resting . He had a terrible longing for some distraction , but he did not know what to do , what to attempt . A new overwhelming sensation was gaining more and more mastery over him every moment ; this was an immeasurable , almost physical , repulsion for everything su
 rrounding him , an obstinate , malignant feeling of hatred . All who met him were loathsome to him\u2014he loathed their faces , their movements , their gestures . If anyone had addressed him , he felt that he might have spat at him or bitten him . . . . He stopped suddenly , on coming out on the bank of the Little Neva , near the bridge to Vassilyevsky Ostrov . " Why , he lives here , in that house , " he thought , " why , I have not come to Razumihin of my own accord ! Here it 's the same thing over again . . . . Very interesting to know , though ; have I come on purpose or have I simply walked here by chance ? Never mind , I said the day before yesterday that I would go and see him the day after ; well , and so I will ! Besides I really cannot go further now . " He went up to Razumihin 's room on the fifth floor . The latter was at home in his garret , busily writing at the moment , and he opened the door himself . It was four months since they had seen each other . Razumihin was si
 tting in a ragged dressing-gown , with slippers on his bare feet , unkempt , unshaven and unwashed . His face showed surprise . " Is it you ? " he cried . He looked his comrade up and down ; then after a brief pause , he whistled . " As hard up as all that ! Why , brother , you 've cut me out ! " he added , looking at Raskolnikov 's rags . " Come sit down , you are tired , I 'll be bound . " And when he had sunk down on the American leather sofa , which was in even worse condition than his own , Razumihin saw at once that his visitor was ill . " Why , you are seriously ill , do you know that ? " He began feeling his pulse . Raskolnikov pulled away his hand . " Never mind , " he said , " I have come for this : I have no lessons . . . . I wanted , . . . but I do n't really want lessons . . . . " " But I say ! You are delirious , you know ! " Razumihin observed , watching him carefully . " No , I am not . " Raskolnikov got up from the sofa . As he had mounted the stairs to Razumihin 's
  , he had not realised that he would be meeting his friend face to face . Now , in a flash , he knew , that what he was least of all disposed for at that moment was to be face to face with anyone in the wide world . His spleen rose within him . He almost choked with rage at himself as soon as he crossed Razumihin 's threshold . " Good-bye , " he said abruptly , and walked to the door . " Stop , stop ! You queer fish . " " I do n't want to , " said the other , again pulling away his hand . " Then why the devil have you come ? Are you mad , or what ? Why , this is . . . almost insulting ! I wo n't let you go like that . " " Well , then , I came to you because I know no one but you who could help . . . to begin . . . because you are kinder than anyone \u2014 cleverer , I mean , and can judge . . . and now I see that I want nothing . Do you hear ? Nothing at all . . . no one 's services . . . no one 's sympathy . I am by myself . . . alone . Come , that 's enough . Leave me alone . " " Sta
 y a minute , you sweep ! You are a perfect madman . As you like for all I care . I have no lessons , do you see , and I do n't care about that , but there 's a bookseller , Heruvimov\u2014and he takes the place of a lesson . I would not exchange him for five lessons . He 's doing publishing of a kind , and issuing natural science manuals and what a circulation they have ! The very titles are worth the money ! You always maintained that I was a fool , but by Jove , my boy , there are greater fools than I am ! Now he is setting up for being advanced , not that he has an inkling of anything , but , of course , I encourage him . Here are two signatures of the German text\u2014in my opinion , the crudest charlatanism ; it discusses the question , 'Is woman a human being ? ' And , of course , triumphantly proves that she is . Heruvimov is going to bring out this work as a contribution to the woman question ; I am translating it ; he will expand these two and a half signatures into six , we shal
 l make up a gorgeous title half a page long and bring it out at half a rouble . It will do ! He pays me six roubles the signature , it works out to about fifteen roubles for the job , and I 've had six already in advance . When we have finished this , we are going to begin a translation about whales , and then some of the dullest scandals out of the second part of Les Confessions we have marked for translation ; somebody has told Heruvimov , that Rousseau was a kind of Radishchev . You may be sure I do n't contradict him , hang him ! Well , would you like to do the second signature of 'Is woman a human being ? ' If you would , take the German and pens and paper\u2014all those are provided , and take three roubles ; for as I have had six roubles in advance on the whole thing , three roubles come to you for your share . And when you have finished the signature there will be another three roubles for you . And please do n't think I am doing you a service ; quite the contrary , as soon as 
 you came in , I saw how you could help me ; to begin with , I am weak in spelling , and secondly , I am sometimes utterly adrift in German , so that I make it up as I go along for the most part . The only comfort is , that it 's bound to be a change for the better . Though who can tell , maybe it 's sometimes for the worse . Will you take it ? " Raskolnikov took the German sheets in silence , took the three roubles and without a word went out . Razumihin gazed after him in astonishment . But when Raskolnikov was in the next street , he turned back , mounted the stairs to Razumihin 's again and laying on the table the German article and the three roubles , went out again , still without uttering a word . " Are you raving , or what ? " Razumihin shouted , roused to fury at last . " What farce is this ? You 'll drive me crazy too . . . what did you come to see me for , damn you ? " " I do n't want . . . translation , " muttered Raskolnikov from the stairs . " Then what the devil do you
  want ? " shouted Razumihin from above . Raskolnikov continued descending the staircase in silence . " Hey , there ! Where are you living ? " No answer . " Well , confound you then ! " But Raskolnikov was already stepping into the street . On the Nikolaevsky Bridge he was roused to full consciousness again by an unpleasant incident . A coachman , after shouting at him two or three times , gave him a violent lash on the back with his whip , for having almost fallen under his horses ' hoofs . The lash so infuriated him that he dashed away to the railing ( for some unknown reason he had been walking in the very middle of the bridge in the traffic ) . He angrily clenched and ground his teeth . He heard laughter , of course . " Serves him right ! " " A pickpocket I dare say . " " Pretending to be drunk , for sure , and getting under the wheels on purpose ; and you have to answer for him . " " It 's a regular profession , that 's what it is . " But while he stood at the railing , still lo
 oking angry and bewildered after the retreating carriage , and rubbing his back , he suddenly felt someone thrust money into his hand . He looked . It was an elderly woman in a kerchief and goatskin shoes , with a girl , probably her daughter wearing a hat , and carrying a green parasol . " Take it , my good man , in Christ 's name . " He took it and they passed on . It was a piece of twenty copecks . From his dress and appearance they might well have taken him for a beggar asking alms in the streets , and the gift of the twenty copecks he doubtless owed to the blow , which made them feel sorry for him . He closed his hand on the twenty copecks , walked on for ten paces , and turned facing the Neva , looking towards the palace . The sky was without a cloud and the water was almost bright blue , which is so rare in the Neva . The cupola of the cathedral , which is seen at its best from the bridge about twenty paces from the chapel , glittered in the sunlight , and in the pure air eve
 ry ornament on it could be clearly distinguished . The pain from the lash went off , and Raskolnikov forgot about it ; one uneasy and not quite definite idea occupied him now completely . He stood still , and gazed long and intently into the distance ; this spot was especially familiar to him . When he was attending the university , he had hundreds of times\u2014generally on his way home\u2014stood still on this spot , gazed at this truly magnificent spectacle and almost always marvelled at a vague and mysterious emotion it roused in him . It left him strangely cold ; this gorgeous picture was for him blank and lifeless . He wondered every time at his sombre and enigmatic impression and , mistrusting himself , put off finding the explanation of it . He vividly recalled those old doubts and perplexities , and it seemed to him that it was no mere chance that he recalled them now . It struck him as strange and grotesque , that he should have stopped at the same spot as before , as though he 
 actually imagined he could think the same thoughts , be interested in the same theories and pictures that had interested him . . . so short a time ago . He felt it almost amusing , and yet it wrung his heart . Deep down , hidden far away out of sight all that seemed to him now\u2014all his old past , his old thoughts , his old problems and theories , his old impressions and that picture and himself and all , all . . . . He felt as though he were flying upwards , and everything were vanishing from his sight . Making an unconscious movement with his hand , he suddenly became aware of the piece of money in his fist . He opened his hand , stared at the coin , and with a sweep of his arm flung it into the water ; then he turned and went home . It seemed to him , he had cut himself off from everyone and from everything at that moment . Evening was coming on when he reached home , so that he must have been walking about six hours . How and where he came back he did not remember . Undressing ,
  and quivering like an overdriven horse , he lay down on the sofa , drew his greatcoat over him , and at once sank into oblivion . . . . It was dusk when he was waked up by a fearful scream . Good God , what a scream ! Such unnatural sounds , such howling , wailing , grinding , tears , blows and curses he had never heard . He could never have imagined such brutality , such frenzy . In terror he sat up in bed , almost swooning with agony . But the fighting , wailing and cursing grew louder and louder . And then to his intense amazement he caught the voice of his landlady . She was howling , shrieking and wailing , rapidly , hurriedly , incoherently , so that he could not make out what she was talking about ; she was beseeching , no doubt , not to be beaten , for she was being mercilessly beaten on the stairs . The voice of her assailant was so horrible from spite and rage that it was almost a croak ; but he , too , was saying something , and just as quickly and indistinctly , hurryin
 g and spluttering . All at once Raskolnikov trembled ; he recognised the voice\u2014it was the voice of Ilya Petrovitch . Ilya Petrovitch here and beating the landlady ! He is kicking her , banging her head against the steps\u2014that 's clear , that can be told from the sounds , from the cries and the thuds . How is it , is the world topsy-turvy ? He could hear people running in crowds from all the storeys and all the staircases ; he heard voices , exclamations , knocking , doors banging . " But why , why , and how could it be ? " he repeated , thinking seriously that he had gone mad . But no , he heard too distinctly ! And they would come to him then next , " for no doubt . . . it 's all about that . . . about yesterday . . . . Good God ! " He would have fastened his door with the latch , but he could not lift his hand . . . besides , it would be useless . Terror gripped his heart like ice , tortured him and numbed him . . . . But at last all this uproar , after continuing about ten min
 utes , began gradually to subside . The landlady was moaning and groaning ; Ilya Petrovitch was still uttering threats and curses . . . . But at last he , too , seemed to be silent , and now he could not be heard . " Can he have gone away ? Good Lord ! " Yes , and now the landlady is going too , still weeping and moaning . . . and then her door slammed . . . . Now the crowd was going from the stairs to their rooms , exclaiming , disputing , calling to one another , raising their voices to a shout , dropping them to a whisper . There must have been numbers of them\u2014almost all the inmates of the block . " But , good God , how could it be ! And why , why had he come here ! " Raskolnikov sank worn out on the sofa , but could not close his eyes . He lay for half an hour in such anguish , such an intolerable sensation of infinite terror as he had never experienced before . Suddenly a bright light flashed into his room . Nastasya came in with a candle and a plate of soup . Looking at him 
 carefully and ascertaining that he was not asleep , she set the candle on the table and began to lay out what she had brought\u2014bread , salt , a plate , a spoon . " You 've eaten nothing since yesterday , I warrant . You 've been trudging about all day , and you 're shaking with fever . " " Nastasya . . . what were they beating the landlady for ? " She looked intently at him . " Who beat the landlady ? " " Just now . . . half an hour ago , Ilya Petrovitch , the assistant superintendent , on the stairs . . . . Why was he ill-treating her like that , and . . . why was he here ? " Nastasya scrutinised him , silent and frowning , and her scrutiny lasted a long time . He felt uneasy , even frightened at her searching eyes . " Nastasya , why do n't you speak ? " he said timidly at last in a weak voice . " It 's the blood , " she answered at last softly , as though speaking to herself . " Blood ? What blood ? " he muttered , growing white and turning towards the wall . Nastasya still looke
 d at him without speaking . " Nobody has been beating the landlady , " she declared at last in a firm , resolute voice . He gazed at her , hardly able to breathe . " I heard it myself . . . . I was not asleep . . . I was sitting up , " he said still more timidly . " I listened a long while . The assistant superintendent came . . . . Everyone ran out on to the stairs from all the flats . " " No one has been here . That 's the blood crying in your ears . When there 's no outlet for it and it gets clotted , you begin fancying things . . . . Will you eat something ? " He made no answer . Nastasya still stood over him , watching him . " Give me something to drink . . . Nastasya . " She went downstairs and returned with a white earthenware jug of water . He remembered only swallowing one sip of the cold water and spilling some on his neck . Then followed forgetfulness . 
\ No newline at end of file


[44/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/GoogleAutoCompleteQueryRunner.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/GoogleAutoCompleteQueryRunner.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/GoogleAutoCompleteQueryRunner.java
new file mode 100644
index 0000000..8cea2d8
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/GoogleAutoCompleteQueryRunner.java
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.similarity.apps;
+
+import java.io.UnsupportedEncodingException;
+import java.net.URLEncoder;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+import org.apache.commons.lang.StringUtils;
+
+import opennlp.tools.similarity.apps.utils.PageFetcher;
+
+public class GoogleAutoCompleteQueryRunner {
+	protected PageFetcher pageFetcher = new PageFetcher();
+	private static String searchRequest = "http://google.com/complete/search?q=",
+			suffix = "&output=toolbar";
+	
+	
+	public List<String> getAutoCompleteExpression(String rawExpr){
+		// insert spaces into camel cases
+		rawExpr= rawExpr.replaceAll("([a-z][a-z])([A-Z][a-z])", "$1 $2");
+		String query = rawExpr.replace(' ', '+');
+		try {
+			query = URLEncoder.encode(query, "UTF-8");
+		} catch (UnsupportedEncodingException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		}
+		
+		String pageOrigHTML = pageFetcher.fetchOrigHTML(searchRequest +query+suffix);
+		String[] results = StringUtils.substringsBetween(pageOrigHTML, "<CompleteSuggestion>", "</CompleteSuggestion>");
+		List<List<String>> accum = new ArrayList<List<String>>();
+		if (results==null)
+				return null;
+		for(String wrapped: results){
+			List<String> accumCase = new ArrayList<String>();
+			String[] words = null;
+			try {
+				words = StringUtils.substringBetween(wrapped, "\"").split(" ");
+			} catch (Exception e){
+				
+			}
+			if (words==null || words.length<1)
+				continue;
+			accumCase = Arrays.asList(words);
+			accum.add(accumCase);
+		}
+		
+		//TODO make more noise-resistant algo
+		if (accum.size()>1){
+			List<String> first = new ArrayList<String>(accum.get(0));
+			List<String> second = new ArrayList<String>(accum.get(1));
+			
+			first.retainAll(second);
+			if (first.size()>0)
+			   return first;
+			else
+			   return accum.get(0);
+		} 
+		
+		if (accum.size()==1)
+			return accum.get(0);
+		
+		
+		return null;
+	}
+	
+	public static String handleCamelCases(String input){
+		String s = input.replaceAll("([a-z,A-Z][a-z])([A-Z][a-z])", "$1 $2").replaceAll("([a-z,A-Z][a-z])([A-Z][a-z])", "$1 $2");
+		s = s.replaceAll("([A-Z])([A-Z][a-z])", "$1 $2");
+		return s;
+	}
+	
+	public static void main(String[] args){
+	
+		
+		String[] tests = new String[]{"SharingInviteNotification", "SharedByMeSortingOptions", "SharedByMeCurrentSortingOption", "GroupedPrivatelySharedByMe",
+				"StorageMeter", "RecentActivities", "StorageMeter", "SharingInviteNotification", 
+				"RecentActivities", "ImporterSuggestionsPrefABC",
+				"WSItem",
+				"SharingInviteNotification",
+				"UserDesktopDevices",
+				"RootFoldersPaginated",
+				"SharingInviteNotification", "apply security settings"};
+		for(String s: tests){
+			System.out.println(handleCamelCases(s));
+		}
+		
+		GoogleAutoCompleteQueryRunner runner = new GoogleAutoCompleteQueryRunner();
+		List<String> 
+		res = runner.getAutoCompleteExpression("commentcount");
+		System.out.println(res);
+		res = runner.getAutoCompleteExpression("clearSess");
+		System.out.println(res);
+	    res = runner.getAutoCompleteExpression("ImporterSuggestionsPref");
+		System.out.println(res);
+	    res = runner.getAutoCompleteExpression("breadCrumbs");
+		System.out.println(res);
+		res = runner.getAutoCompleteExpression("RootFolder");
+		System.out.println(res);
+		
+		res = runner.getAutoCompleteExpression("BreadCrumbList");
+		System.out.println(res);
+		
+	}
+
+}
+

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/SentenceTranslate.java.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/SentenceTranslate.java.txt b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/SentenceTranslate.java.txt
new file mode 100644
index 0000000..414b8d1
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/SentenceTranslate.java.txt
@@ -0,0 +1,212 @@
+package opennlp.tools.similarity.apps;
+
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.logging.Logger;
+
+import opennlp.tools.parse_thicket.ParseCorefsBuilder;
+import opennlp.tools.parse_thicket.ParseThicket;
+import opennlp.tools.parse_thicket.apps.SnippetToParagraph;
+import opennlp.tools.parse_thicket.matching.Matcher;
+import opennlp.tools.similarity.apps.BingQueryRunner;
+import opennlp.tools.similarity.apps.Fragment;
+import opennlp.tools.similarity.apps.GeneratedSentenceProcessor;
+import opennlp.tools.similarity.apps.HitBase;
+import opennlp.tools.similarity.apps.RelatedSentenceFinder;
+import opennlp.tools.similarity.apps.utils.PageFetcher;
+import opennlp.tools.similarity.apps.utils.Utils;
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+import opennlp.tools.textsimilarity.ParseTreeChunkListScorer;
+import opennlp.tools.textsimilarity.SentencePairMatchResult;
+import opennlp.tools.textsimilarity.chunker2matcher.ParserChunker2MatcherProcessor;
+
+import com.memetix.mst.language.Language;
+import com.memetix.mst.translate.Translate;
+
+import edu.stanford.nlp.ling.Sentence;
+import edu.stanford.nlp.ling.TaggedWord;
+import edu.stanford.nlp.trees.Tree;
+import edu.stanford.nlp.trees.tregex.TregexMatcher;
+import edu.stanford.nlp.trees.tregex.TregexPattern;
+
+/**
+ * Class for sentence translation with improvement
+ * based on parse thickets.
+ * @author Alex Loptev
+ *
+ */
+public class SentenceTranslate {
+        private static String clientID = "ParseThicketsTranslation";
+        private static String clientSecret = "M4teDPWKv5xMTOZ/v6nJbwya4ilPE0cUCK4cCPGeRok=";
+
+        private static Logger LOG;
+        private static ParseTreeChunkListScorer parseTreeChunkListScorer; 
+        private static Matcher matcher;
+        private static BingQueryRunner searchRunner;
+        private static SnippetToParagraph sentenceRetriever; 
+        private static final int CONSIDERABLE_SEARCH_RESULTS_COUNT = 1;
+        private static final double MEANINGLESS_THRESHOLD = 3.0;
+        private static final int MINIMUM_WORDS_IN_PHRASE_FOR_TESTING = 5;
+        
+        /**
+         * Substitutes translation fragments by text fragments found on the Web with enough
+         * similarity score with original translation.
+         * @param translatedSentence sentence translated by some translator (e.g. Microsoft Translator)
+         * @return improvedTranslation
+         */
+        public static String improveSentenceTranslationBySimilartyAssessment(String translatedSentence) {
+                List<Tree> phraseNodesForTesting = formPhrasesForMeaningfulnessTesting(translatedSentence);
+                for (Tree phraseNode: phraseNodesForTesting) {
+                        String phrase = Sentence.listToString(phraseNode.yield());
+                        String quotedPhrase = "\"" + phrase + "\"";
+                        HitBase mostSimilarResult = null;
+                        double mostSimilarScore = 0.0;
+                        boolean meaningfull = false;
+                        String[] phrases = {quotedPhrase, phrase};
+                        for (String p: Arrays.asList(phrases)) {
+                                LOG.info(String.format("Meaningfulness testing for phrase: %s", p));
+                                List<HitBase> searchResults = searchRunner.runSearch(p, CONSIDERABLE_SEARCH_RESULTS_COUNT);
+                                for (HitBase searchResult: searchResults) {
+                                        double score = assessSimilarityWithHitBase(phrase, searchResult);
+                                        if (score > MEANINGLESS_THRESHOLD ) {
+                                                meaningfull = true;
+                                                LOG.info(String.format("Phrase %s is meaningful. Score is %f", phrase, score));
+                                                break;
+                                        }
+                                        if (mostSimilarScore < score) {
+                                                mostSimilarResult = searchResult;
+                                                mostSimilarScore = score;
+                                        }
+                                }
+                                if (meaningfull)
+                                        break;
+                        }
+                        if (!meaningfull) {
+                                LOG.info(String.format("Phrase %s is meaningless. Maximal score is %f", phrase, mostSimilarScore));
+                                // TODO: replacing meaningless phrase
+                        }
+                }
+                return "";
+        }
+        
+        public static double assessSimilarityScore(String s1, String s2) {
+                LOG.info(String.format("Assess similarity between: \"%s\" and \"%s\"", s1, s2));
+                List<List<ParseTreeChunk>> match = matcher.assessRelevance(s1, s2);
+                double sim = parseTreeChunkListScorer.getParseTreeChunkListScore(match);
+                LOG.info(String.format("Score: %f", sim));
+                return sim;
+        }
+        
+        /**
+         * Assesses similarity score for phrase and search result's:
+         * title, snippet and appropriate document sentence.
+         * @param sentence
+         * @param searchResult
+         * @return similarity score
+         */
+        private static double assessSimilarityWithHitBase(String phrase, HitBase searchResult) {
+                String title = searchResult.getTitle().replace("<b>", " ").replace("</b>", " ").replace("  ", " ").replace("  ", " ");
+                String snippet = searchResult.getAbstractText().replace("<b>", " ").replace("</b>", " ").replace("  ", " ").replace("  ", " ");
+                double score = Math.max(assessSimilarityScore(phrase, title), assessSimilarityScore(phrase, snippet));
+                searchResult = sentenceRetriever.formTextFromOriginalPageGivenSnippet(searchResult);
+                List<String> sentences = searchResult.getOriginalSentences();
+                for (String sentence: sentences) {
+                        score = Math.max(score, assessSimilarityScore(phrase, sentence));
+                }
+            return score;
+        }
+        
+        /**
+         * Creates list of phrases (L_op) from translated sentence for meaningfulness testing.
+         * Such list includes all the phrases which contain at least two sub-phrases. 
+         * @param sentence
+         * @return list of phrases containing at least two sub-phrases
+         */
+        private static List<Tree> formPhrasesForMeaningfulnessTesting(String sentence) {
+                List<Tree> results = new LinkedList<Tree>();
+                ParseCorefsBuilder ptBuilder = ParseCorefsBuilder.getInstance();
+                ParseThicket pt = ptBuilder.buildParseThicket(sentence);
+                Tree t = pt.getSentences().get(0);
+                // tregex pattern for all nodes with at least two phrasal children
+                TregexPattern pattern = TregexPattern.compile("__ < (__ [ !<: __ | < (__ < __) ] $ (__ !<: __ | < (__ < __)))");
+                TregexMatcher matcher = pattern.matcher(t);
+                while (matcher.findNextMatchingNode()) {
+                        Tree candidate = matcher.getMatch();
+                        int wordsCount = 0;
+                        // test if phrase is too short
+                        for (TaggedWord leaf: candidate.taggedYield()) {
+                                // if is not punctuation
+                                if (Character.isLetter(leaf.tag().charAt(0))) {
+                                        wordsCount++;
+                                }
+                        }
+                        if (wordsCount >= MINIMUM_WORDS_IN_PHRASE_FOR_TESTING) {
+                                results.add(candidate);
+                        }
+                }
+                // reversing phrases because the highest nodes in tree should 
+                // be tested for meaningfulness after the lowest nodes
+                Collections.reverse(results);
+                return results;
+        }
+        
+        
+        /**
+         * Execute simple sentence translation by Microsoft Translation API.
+         * 
+         * @param        sentence                sentence for translation
+         * @param        fromLanguage        sentence native language
+         * @param        fromLanguage        sentence destination language
+         * @return        translated                 sentence
+         * @throws Exception 
+         */
+        public static String executeByMicrosoftTranslator(String text, Language fromLanguage, Language toLanguage) throws Exception {
+                String result = Translate.execute(text, fromLanguage, toLanguage);
+                LOG.info(text + " -> " + result);
+            return result;
+        }
+        
+        /**
+         * Execute simple sentence translation to English by Microsoft Translation API
+         * with sentence native language auto detection.
+         * 
+         * @param        sentence                sentence for translation
+         * @return        translated                 sentence
+         * @throws Exception 
+         */
+        public static String executeByMicrosoftTranslator(String text) throws Exception {
+                return executeByMicrosoftTranslator(text, Language.AUTO_DETECT, Language.ENGLISH);
+        }
+        
+        public static void setMicrosoftTranslatorClientId(String clientId) {
+                Translate.setClientId(clientId);
+        }
+        
+        public static void setMicrosoftTranslatorClientSecret(String clientSecret) {
+                Translate.setClientSecret(clientSecret);
+        }
+        
+        /**
+         * Static initialization block.
+         */
+        static {
+                Translate.setClientId(clientID);
+            Translate.setClientSecret(clientSecret);
+            searchRunner = new BingQueryRunner();
+            sentenceRetriever = new SnippetToParagraph();
+            matcher = new Matcher();
+            parseTreeChunkListScorer = new ParseTreeChunkListScorer();
+            LOG = Logger.getLogger("opennlp.tools.parse_thicket.translation.SentenceTranslate");
+        }
+        
+        /**
+         * Dummy method for testing purposes.
+         * @param args
+         */
+        public static void main(String[] args) throws Exception {
+                SentenceTranslate.improveSentenceTranslationBySimilartyAssessment(SentenceTranslate.executeByMicrosoftTranslator("\ufffd \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd, 8 \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd, \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd \ufffd\ufffd\ufffd\ufffd\ufffd \ufffd\ufffd\ufffd\ufffd\ufffd \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd \ufffd \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd \ufffd\ufffd\ufffd\ufffd\ufffd \ufffd\ufffd\ufffd\ufffd \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd, \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd \ufffd \ufffd\ufffd\ufffd\ufffd \ufffd\ufffd\ufffd\ufffd \ufffd\ufffd\ufffd\ufffd\ufffd \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd."));
+        }}
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/Comment.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/Comment.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/Comment.java
new file mode 100644
index 0000000..91ba23f
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/Comment.java
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.similarity.apps.solr;
+
+import java.util.Iterator;
+import java.util.List;
+
+import org.docx4j.jaxb.Context;
+import org.docx4j.openpackaging.exceptions.InvalidFormatException;
+import org.docx4j.openpackaging.packages.WordprocessingMLPackage;
+import org.docx4j.openpackaging.parts.JaxbXmlPart;
+import org.docx4j.openpackaging.parts.WordprocessingML.CommentsPart;
+import org.docx4j.openpackaging.parts.WordprocessingML.MainDocumentPart;
+import org.docx4j.relationships.Relationship;
+import org.docx4j.relationships.Relationships;
+import org.docx4j.wml.Comments;
+import org.docx4j.wml.ObjectFactory;
+
+//import util.RelationshipName;
+
+public class Comment {
+
+    private WordprocessingMLPackage wordMlPackage;
+    private boolean relSet = false;
+
+    public Comment(WordprocessingMLPackage wordMLPack) {
+        this.wordMlPackage = wordMLPack;
+        setCommentRel();
+    }
+
+    private void setCommentRel() {
+        if (!commentRelSet()) {
+            CommentsPart cp;
+            try {
+                cp = new CommentsPart();
+                // Part must have minimal contents
+                org.docx4j.wml.ObjectFactory wmlObjectFactory = new ObjectFactory();
+                wordMlPackage.getMainDocumentPart().addTargetPart(cp);
+            } catch (InvalidFormatException e) {
+                // TODO Auto-generated catch block
+                e.printStackTrace();
+            }
+        }
+    }
+
+    private boolean commentRelSet() {
+        Relationship relShip;
+        boolean relSet = false;
+        if (!relSet) {
+            JaxbXmlPart<Relationships> jxpRelShips = wordMlPackage
+                    .getMainDocumentPart().getOwningRelationshipPart();
+            Relationships pk = jxpRelShips.getJaxbElement();
+
+            List<Relationship> mc = pk.getRelationship();
+
+            Iterator<Relationship> it = mc.iterator();
+       /*     while (it.hasNext() && !relSet) {
+                relShip = it.next();
+                if (relShip.getValue().equalsIgnoreCase(
+                        RelationshipName.commentIdentifier)) {
+                    relSet = true;
+                }
+            }*/
+        }
+        return relSet;
+    }
+    
+    public static void main(String[] args) throws Exception {
+
+        // Create a package
+        WordprocessingMLPackage wmlPack = new WordprocessingMLPackage();
+
+        // Create main document part
+        MainDocumentPart wordDocumentPart = new MainDocumentPart();      
+        
+        // Create main document part content
+        org.docx4j.wml.ObjectFactory factory = Context.getWmlObjectFactory();
+        org.docx4j.wml.Body  body = factory.createBody();      
+        org.docx4j.wml.Document wmlDocumentEl = factory.createDocument();
+        
+        wmlDocumentEl.setBody(body);
+        wordDocumentPart.setJaxbElement(wmlDocumentEl);
+        wmlPack.addTargetPart(wordDocumentPart);
+        
+        CommentsPart cp = new CommentsPart();
+        // Part must have minimal contents
+        Comments comments = factory.createComments();
+        cp.setJaxbElement(comments);
+        
+        wordDocumentPart.addTargetPart(cp);
+        
+        // Now you can add comments to your comments part,
+        // and comment refs in your main document part   
+        
+        
+        
+        
+        
+        
+              
+        wmlPack.save(new java.io.File(System.getProperty("user.dir")+ "/out-m.docx"));       
+      }
+    
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/CommentsRel.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/CommentsRel.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/CommentsRel.java
new file mode 100644
index 0000000..fe3c14f
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/CommentsRel.java
@@ -0,0 +1,139 @@
+package opennlp.tools.similarity.apps.solr;
+
+import java.io.File;
+import java.io.IOException;
+import java.math.BigInteger;
+
+import javax.xml.bind.JAXBException;
+
+import org.docx4j.XmlUtils;
+import org.docx4j.jaxb.Context;
+import org.docx4j.openpackaging.exceptions.Docx4JException;
+import org.docx4j.openpackaging.exceptions.InvalidFormatException;
+import org.docx4j.openpackaging.packages.WordprocessingMLPackage;
+import org.docx4j.openpackaging.parts.WordprocessingML.CommentsPart;
+import org.docx4j.openpackaging.parts.WordprocessingML.EndnotesPart;
+import org.docx4j.openpackaging.parts.WordprocessingML.MainDocumentPart;
+import org.docx4j.wml.CTEndnotes;
+import org.docx4j.wml.CTFtnEdn;
+import org.docx4j.wml.Comments;
+
+public class CommentsRel {
+	   
+	   private WordprocessingMLPackage wordMlPackage;
+	   private MainDocumentPart mainPart;
+	   private boolean relSet = false;
+	   private org.docx4j.wml.ObjectFactory wmlObjectFactory;
+	   private CommentsPart cmPart;
+
+	   
+	   public CommentsRel(WordprocessingMLPackage wordMLPack) {
+	      this.wordMlPackage = wordMLPack;
+	      wmlObjectFactory = new org.docx4j.wml.ObjectFactory();
+	      setCommentRel();
+	      cmPart = wordMlPackage.getMainDocumentPart().getCommentsPart();
+	      mainPart = wordMLPack.getMainDocumentPart();
+	   }
+
+	   private void setCommentRel() {
+	      if (!commentRelSet()) {
+	         CommentsPart cp;
+	         try {
+	            cp = new CommentsPart();
+	            // Part must have minimal contents
+	            Comments comments = wmlObjectFactory.createComments();
+	               cp.setJaxbElement(comments);            
+	            
+	            wordMlPackage.getMainDocumentPart().addTargetPart(cp);
+	         } catch (InvalidFormatException e) {
+	            // TODO Auto-generated catch block
+	            e.printStackTrace();
+	         }
+	      }
+	   }
+
+	   private boolean commentRelSet() {
+	      return !(wordMlPackage.getMainDocumentPart().getCommentsPart() == null);
+	   }
+
+	   public void addNewComment(String author, String text) {
+	      
+//	      XMLGregorianCalendar xmlCal = new XMLGregorianCalendarImpl();  // You'll need to fix this!
+	      
+//	      CommentRangeEnd cRangeEnde = wmlObjectFactory.createCommentRangeEnd();
+//	      CommentRangeStart cRangeStart = wmlObjectFactory
+//	            .createCommentRangeStart();
+	      Comments comment = wmlObjectFactory.createComments();
+//	      Comments.Comment myCom = wm
+
+	      org.docx4j.wml.Comments.Comment c = Context.getWmlObjectFactory().createCommentsComment();
+	      System.out.println("test");
+//	      comment.setParent(cmPart);
+	      c.setAuthor(author);
+//	      c.setDate(xmlCal);
+	      cmPart.getJaxbElement().getComment().add(c);
+
+	      System.out.println("test ende");
+	   }
+
+	//   WordprocessingMLPackage wordML;
+
+	   public static void main(String args[]) throws IOException {
+
+	      File document = new File("C:/workspace/TestSolr/mydoc.docx");
+
+	      MainDocumentPart mDocPart;
+	      try {
+	        /* mlPackage = new WordprocessingMLPackage().load(new File(document.getCanonicalPath()));
+
+	         mDocPart = mlPackage.getMainDocumentPart();
+
+	         CommentsRel myComment = new CommentsRel(mlPackage);
+
+	         myComment.addNewComment("MC","Text");
+	         */
+	      // Add an endnote
+	         
+	         WordprocessingMLPackage mlPackage = WordprocessingMLPackage.createPackage();
+	         
+	         // Setup endnotes part
+	         EndnotesPart ep = new EndnotesPart();
+	         CTEndnotes endnotes = Context.getWmlObjectFactory().createCTEndnotes();
+	         ep.setJaxbElement(endnotes);
+	         mlPackage.getMainDocumentPart().addTargetPart(ep);
+	         
+	         CTFtnEdn endnote = Context.getWmlObjectFactory().createCTFtnEdn();
+	         endnotes.getEndnote().add(endnote);
+	         
+	         endnote.setId(BigInteger.ONE.add(BigInteger.ONE));
+	         String endnoteBody = "<w:p xmlns:w=\"http://schemas.openxmlformats.org/wordprocessingml/2006/main\" ><w:pPr><w:pStyle w:val=\"EndnoteText\"/></w:pPr><w:r><w:rPr><w:rStyle w:val=\"EndnoteReference\"/></w:rPr><w:endnoteRef/></w:r><w:r><w:t xml:space=\"preserve\"> An endnote</w:t></w:r></w:p>";
+	         try {
+				endnote.getEGBlockLevelElts().add( XmlUtils.unmarshalString(endnoteBody));
+			} catch (JAXBException e) {
+				// TODO Auto-generated catch block
+				e.printStackTrace();
+			}
+	         
+	         // Add the body text referencing it
+	         String docBody = "<w:p xmlns:w=\"http://schemas.openxmlformats.org/wordprocessingml/2006/main\" ><w:r><w:t>the quick brown</w:t></w:r><w:r><w:rPr><w:rStyle w:val=\"EndnoteReference\"/></w:rPr><w:endnoteReference w:id=\"2\"/></w:r></w:p>";
+	         
+	         try {
+				mlPackage.getMainDocumentPart().addParagraph(docBody);
+			} catch (JAXBException e) {
+				// TODO Auto-generated catch block
+				e.printStackTrace();
+			}
+	         
+	         
+	         
+	         
+	         
+	         
+	          mlPackage.save(new File("C:/workspace/TestSolr/mydoc.docx-OUT.docx"));
+	      } catch (Docx4JException e) {
+	         // TODO Auto-generated catch block
+	         e.printStackTrace();
+	      }
+
+	   }
+	}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/QueryExpansionRequestHandler.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/QueryExpansionRequestHandler.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/QueryExpansionRequestHandler.java
new file mode 100644
index 0000000..e98e6be
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/QueryExpansionRequestHandler.java
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.similarity.apps.solr;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+import opennlp.tools.similarity.apps.HitBaseComparable;
+import opennlp.tools.similarity.apps.utils.Pair;
+import opennlp.tools.textsimilarity.ParseTreeChunkListScorer;
+import opennlp.tools.textsimilarity.SentencePairMatchResult;
+import opennlp.tools.textsimilarity.chunker2matcher.ParserChunker2MatcherProcessor;
+
+import org.apache.commons.lang.ArrayUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.index.CorruptIndexException;
+import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.queryparser.classic.ParseException;
+import org.apache.lucene.search.BooleanClause.Occur;
+import org.apache.lucene.search.BooleanQuery;
+import org.apache.lucene.search.CachingWrapperFilter;
+import org.apache.lucene.search.Collector;
+import org.apache.lucene.search.Filter;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.QueryWrapperFilter;
+import org.apache.lucene.search.ScoreDoc;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CommonParams;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.handler.RequestHandlerBase;
+import org.apache.solr.handler.component.ResponseBuilder;
+import org.apache.solr.handler.component.SearchComponent;
+import org.apache.solr.handler.component.SearchHandler;
+import org.apache.solr.handler.component.ShardHandler;
+import org.apache.solr.handler.component.ShardRequest;
+import org.apache.solr.handler.component.ShardResponse;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.ResultContext;
+import org.apache.solr.response.SolrQueryResponse;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.DocSlice;
+import org.apache.solr.search.QParser;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.util.RTimer;
+import org.apache.solr.util.SolrPluginUtils;
+
+public class QueryExpansionRequestHandler extends SearchHandler {
+
+	public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse rsp){
+		try {
+			//System.out.println("request before ="+req);
+			SolrQueryRequest req1 = substituteField(req);
+			//System.out.println("request after ="+req1);
+			super.handleRequestBody(req1, rsp);
+		} catch (Exception e) {
+			e.printStackTrace();
+		}
+		
+	}
+
+	public static SolrQueryRequest substituteField(SolrQueryRequest req){
+		SolrParams params = req.getParams();
+		String query = params.get("q");
+		System.out.println("query before ="+query);
+		query = query.replace(' ', '_');
+		System.out.println("query after ="+query);
+		NamedList values = params.toNamedList();
+		values.remove("q");
+		values.add("q", query);
+		params = SolrParams.toSolrParams(values);
+		req.setParams(params);
+		return req;
+	}
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/SearchResultsReRankerStanfRequestHandler.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/SearchResultsReRankerStanfRequestHandler.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/SearchResultsReRankerStanfRequestHandler.java
new file mode 100644
index 0000000..477f022
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/SearchResultsReRankerStanfRequestHandler.java
@@ -0,0 +1,306 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.similarity.apps.solr;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.logging.Logger;
+
+import opennlp.tools.parse_thicket.apps.SnippetToParagraph;
+import opennlp.tools.parse_thicket.matching.Matcher;
+import opennlp.tools.similarity.apps.BingQueryRunner;
+import opennlp.tools.similarity.apps.HitBase;
+import opennlp.tools.similarity.apps.HitBaseComparable;
+import opennlp.tools.similarity.apps.utils.Pair;
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+import opennlp.tools.textsimilarity.ParseTreeChunkListScorer;
+import opennlp.tools.textsimilarity.SentencePairMatchResult;
+import opennlp.tools.textsimilarity.chunker2matcher.ParserChunker2MatcherProcessor;
+
+import org.apache.commons.lang.ArrayUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.lucene.document.Document;
+import org.apache.lucene.index.CorruptIndexException;
+import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.queryparser.classic.ParseException;
+import org.apache.lucene.search.BooleanClause.Occur;
+import org.apache.lucene.search.BooleanQuery;
+import org.apache.lucene.search.CachingWrapperFilter;
+import org.apache.lucene.search.Collector;
+import org.apache.lucene.search.Filter;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.QueryWrapperFilter;
+import org.apache.lucene.search.ScoreDoc;
+import org.apache.solr.common.SolrDocument;
+import org.apache.solr.common.SolrDocumentList;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.params.CommonParams;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.common.params.ShardParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.handler.component.SearchHandler;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.response.SolrQueryResponse;
+
+
+
+public class SearchResultsReRankerStanfRequestHandler extends SearchHandler {
+	private static Logger LOG = Logger
+			.getLogger("com.become.search.requestHandlers.SearchResultsReRankerRequestHandler");
+	private final static int MAX_SEARCH_RESULTS = 100;
+	private ParseTreeChunkListScorer parseTreeChunkListScorer = new ParseTreeChunkListScorer();
+	private int MAX_QUERY_LENGTH_NOT_TO_RERANK=3;
+	private Matcher matcher = new Matcher();
+	private BingQueryRunner bingSearcher = new BingQueryRunner();
+	private SnippetToParagraph snp = new SnippetToParagraph();
+	
+	
+	public void handleRequestBody(SolrQueryRequest req, SolrQueryResponse rsp){
+		// get query string
+		String requestExpression = req.getParamString();
+		String[] exprParts = requestExpression.split("&");
+		for(String part: exprParts){
+			if (part.startsWith("q="))
+				requestExpression = part;			
+		}
+		String query = StringUtils.substringAfter(requestExpression, ":");
+		LOG.info(requestExpression);
+
+
+		SolrParams ps = req.getOriginalParams();
+		Iterator<String> iter =  ps.getParameterNamesIterator();
+		List<String> keys = new ArrayList<String>();
+		while(iter.hasNext()){
+			keys.add(iter.next());
+		}
+
+		List<HitBase> searchResults = new ArrayList<HitBase>();
+
+
+
+
+
+		for ( Integer i=0; i< MAX_SEARCH_RESULTS; i++){
+			String title = req.getParams().get("t"+i.toString());
+			String descr = req.getParams().get("d"+i.toString());
+
+			if(title==null || descr==null)
+				continue;
+
+			HitBase hit = new HitBase();
+			hit.setTitle(title);
+			hit.setAbstractText(descr);
+			hit.setSource(i.toString());
+			searchResults.add(hit);
+		}
+
+		/*
+		 * http://173.255.254.250:8983/solr/collection1/reranker/?
+		 * q=search_keywords:design+iphone+cases&fields=spend+a+day+with+a+custom+iPhone+case&fields=Add+style+to+your+every+day+fresh+design+with+a+custom+iPhone+case&fields=Add+style+to+your+every+day+with+mobile+case+for+your+family&fields=Add+style+to+your+iPhone+and+iPad&fields=Add+Apple+fashion+to+your+iPhone+and+iPad
+		 * 
+		 */
+
+		if (searchResults.size()<1) {
+			int count=0;
+			for(String val : exprParts){
+				if (val.startsWith("fields=")){
+					val = StringUtils.mid(val, 7, val.length());
+					HitBase hit = new HitBase();
+					hit.setTitle("");
+					hit.setAbstractText(val);
+					hit.setSource(new Integer(count).toString());
+					searchResults.add(hit);
+					count++;
+				}
+
+			}
+		}
+
+
+		List<HitBase> reRankedResults = null;
+		query = query.replace('+', ' ');
+		if (tooFewKeywords(query)|| orQuery(query)){
+			reRankedResults = searchResults;
+			LOG.info("No re-ranking for "+query);
+		}
+		else 
+			reRankedResults = calculateMatchScoreResortHits(searchResults, query);
+		/*
+		 * <scores>
+<score index="2">3.0005</score>
+<score index="1">2.101</score>
+<score index="3">2.1003333333333334</score>
+<score index="4">2.00025</score>
+<score index="5">1.1002</score>
+</scores>
+		 * 
+		 * 
+		 */
+		StringBuffer buf = new StringBuffer(); 
+		buf.append("<scores>");
+		for(HitBase hit: reRankedResults){
+			buf.append("<score index=\""+hit.getSource()+"\">"+hit.getGenerWithQueryScore()+"</score>");				
+		}
+		buf.append("</scores>");
+
+		NamedList<Object> scoreNum = new NamedList<Object>();
+		for(HitBase hit: reRankedResults){
+			scoreNum.add(hit.getSource(), hit.getGenerWithQueryScore());				
+		}
+		
+		StringBuffer bufNums = new StringBuffer(); 
+		bufNums.append("order>");
+		for(HitBase hit: reRankedResults){
+			bufNums.append(hit.getSource()+"_");				
+		}
+		bufNums.append("/order>");
+		
+		LOG.info("re-ranking results: "+buf.toString());
+		NamedList<Object> values = rsp.getValues();
+		values.remove("response");
+		values.add("response", scoreNum); 
+		values.add("new_order", bufNums.toString().trim());
+		rsp.setAllValues(values);
+		
+	}
+
+	private boolean orQuery(String query) {
+		if (query.indexOf('|')>-1)
+			return true;
+
+		return false;
+	}
+
+	private boolean tooFewKeywords(String query) {
+		String[] parts = query.split(" ");
+		if (parts!=null && parts.length< MAX_QUERY_LENGTH_NOT_TO_RERANK)
+			return true;
+
+		return false;
+	}
+
+	protected List<HitBase> calculateMatchScoreResortHits(List<HitBase> hits,
+			String searchQuery) {
+
+		List<HitBase> newHitList = new ArrayList<HitBase>();
+		int count = 0;
+		for (HitBase hit : hits) {
+			if (count>10)
+				break;
+			count++;
+			String[] pageSentsAndSnippet = formTextForReRankingFromHit(hit);
+					
+			Double score = 0.0;
+			try {
+				List<List<ParseTreeChunk>> match = null;
+				if (pageSentsAndSnippet!=null && pageSentsAndSnippet[0].length()>50){
+					match = matcher.assessRelevanceCache(pageSentsAndSnippet[0] ,
+							searchQuery);
+					score = parseTreeChunkListScorer.getParseTreeChunkListScore(match);
+					hit.setSource(match.toString());
+				}
+				if (score < 2){ // attempt to match with snippet, if not much luck with original text
+					match = matcher.assessRelevanceCache(pageSentsAndSnippet[0] ,
+							searchQuery);
+					score = parseTreeChunkListScorer.getParseTreeChunkListScore(match);
+				}
+				LOG.info(score + " | " +pageSentsAndSnippet[1]);
+			} catch (Exception e) {
+				LOG.severe("Problem processing snapshot " + pageSentsAndSnippet[1]);
+				e.printStackTrace();
+			}
+			hit.setGenerWithQueryScore(score);
+			newHitList.add(hit);
+		}
+		
+		System.out.println("\n\n ============= old ORDER ================= ");
+		for (HitBase hit : newHitList) {
+			System.out.println(hit.getOriginalSentences().toString() + " => "+hit.getGenerWithQueryScore());
+			System.out.println("match = "+hit.getSource());
+		}
+		Collections.sort(newHitList, new HitBaseComparable());
+
+		System.out.println("\n\n ============= NEW ORDER ================= ");
+		for (HitBase hit : newHitList) {
+			System.out.println(hit.getOriginalSentences().toString() + " => "+hit.getGenerWithQueryScore());
+			System.out.println("match = "+hit.getSource());
+		}
+
+		return newHitList;
+	}
+	
+	protected String[] formTextForReRankingFromHit(HitBase hit) {
+		HitBase hitWithFullSents = snp.formTextFromOriginalPageGivenSnippet(hit);
+		String textFromOriginalPage = "";
+		try {
+			List<String> sents = hitWithFullSents.getOriginalSentences();
+			for(String s: sents){
+				textFromOriginalPage+=s+" ";
+			}
+
+			if (textFromOriginalPage.startsWith(".")){
+				textFromOriginalPage = textFromOriginalPage.substring(2);
+			}
+			textFromOriginalPage = textFromOriginalPage.replace(" . .", ". ").replace(". . ", ". ").
+					replace("..", ". ").trim();
+		} catch (Exception e1) {
+			e1.printStackTrace();
+			LOG.info("Problem processing snapshot "+hit.getAbstractText());
+		}
+		hit.setPageContent(textFromOriginalPage);
+		String snapshot = hit.getAbstractText().replace("<b>...</b>", ". ").replace("<span class='best-phrase'>", " ").replace("<span>", " ").replace("<span>", " ")
+				.replace("<b>", "").replace("</b>", "");
+		snapshot = snapshot.replace("</B>", "").replace("<B>", "")
+				.replace("<br>", "").replace("</br>", "").replace("...", ". ")
+				.replace("|", " ").replace(">", " ").replace(". .", ". ");
+		snapshot += " . " + hit.getTitle();
+		
+		return new String[] { textFromOriginalPage, snapshot };
+	}
+
+
+	public class HitBaseComparable implements Comparator<HitBase> {
+		// @Override
+		public int compare(HitBase o1, HitBase o2) {
+			return (o1.getGenerWithQueryScore() > o2.getGenerWithQueryScore() ? -1
+					: (o1 == o2 ? 0 : 1));
+		}
+	}
+
+}
+
+/*
+
+http://dev1.exava.us:8086/solr/collection1/reranker/?q=search_keywords:I+want+style+in+my+every+day+fresh+design+iphone+cases
+&t1=Personalized+iPhone+4+Cases&d1=spend+a+day+with+a+custom+iPhone+case
+&t2=iPhone+Cases+to+spend+a+day&d2=Add+style+to+your+every+day+fresh+design+with+a+custom+iPhone+case
+&t3=Plastic+iPhone+Cases&d3=Add+style+to+your+every+day+with+mobile+case+for+your+family
+&t4=Personalized+iPhone+and+iPad+Cases&d4=Add+style+to+your+iPhone+and+iPad
+&t5=iPhone+accessories+from+Apple&d5=Add+Apple+fashion+to+your+iPhone+and+iPad
+
+http://dev1.exava.us:8086/solr/collection1/reranker/?q=search_keywords:I+want+style+in+my+every+day+fresh+design+iphone+cases&t1=Personalized+iPhone+4+Cases&d1=spend+a+day+with+a+custom+iPhone+case&t2=iPhone+Cases+to+spend+a+day&d2=Add+style+to+your+every+day+fresh+design+with+a+custom+iPhone+case&t3=Plastic+iPhone+Cases&d3=Add+style+to+your+every+day+with+mobile+case+for+your+family&t4=Personalized+iPhone+and+iPad+Cases&d4=Add+style+to+your+iPhone+and+iPad&t5=iPhone+accessories+from+Apple&d5=Add+Apple+fashion+to+your+iPhone+and+iPad
+ */
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/WordDocBuilder.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/WordDocBuilder.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/WordDocBuilder.java
new file mode 100644
index 0000000..3de5c1b
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/WordDocBuilder.java
@@ -0,0 +1,270 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.similarity.apps.solr;
+
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+
+import net.billylieurance.azuresearch.AzureSearchImageResult;
+import net.billylieurance.azuresearch.AzureSearchResultSet;
+import net.billylieurance.azuresearch.AzureSearchWebResult;
+
+import org.docx4j.dml.wordprocessingDrawing.Inline;
+import org.docx4j.openpackaging.exceptions.Docx4JException;
+import org.docx4j.openpackaging.exceptions.InvalidFormatException;
+import org.docx4j.openpackaging.packages.WordprocessingMLPackage;
+import org.docx4j.openpackaging.parts.WordprocessingML.BinaryPartAbstractImage;
+import org.docx4j.wml.CTFootnotes;
+import org.docx4j.wml.CTFtnEdn;
+import org.docx4j.wml.Drawing;
+import org.docx4j.wml.P;
+import org.docx4j.wml.R;
+import org.docx4j.wml.STFtnEdn;
+
+import opennlp.tools.similarity.apps.BingQueryRunner;
+import opennlp.tools.similarity.apps.Fragment;
+import opennlp.tools.similarity.apps.HitBase;
+
+public class WordDocBuilder{
+	protected static final String IMG_REL_PATH = "images/";
+	protected BingQueryRunner imageSearcher = new BingQueryRunner();
+	protected String absPath = null;
+	
+	public WordDocBuilder(){
+	absPath = new File(".").getAbsolutePath();
+	absPath = absPath.substring(0, absPath.length()-1);
+	}
+	
+	public String buildWordDoc(List<HitBase> content, String title){
+		
+		String outputDocFinename =  absPath+"/written/"+ title.replace(' ','_').replace('\"', ' ').trim()+ ".docx";
+		
+		WordprocessingMLPackage wordMLPackage;
+		try {
+			wordMLPackage = WordprocessingMLPackage.createPackage();
+			wordMLPackage.getMainDocumentPart().addStyledParagraphOfText("Title", title);
+			for(HitBase para: content){
+				
+				wordMLPackage.getMainDocumentPart().addStyledParagraphOfText("Subtitle",
+						para.getTitle());
+				String paraText = para.getFragments().toString().replace("[", "").replace("]", "").replace(" | ", "")
+						.replace(".,", ".").replace(".\"", "\"").replace(". .", ".")
+						.replace(",.", ".");
+				wordMLPackage.getMainDocumentPart().addParagraphOfText(paraText);
+				
+				addImageByImageTitleToPackage(wordMLPackage, para.getTitle());
+			}
+			
+			//File file = new File("C:/ma/personal/argCamp.png");
+	        //byte[] bytes = convertImageToByteArray(file);
+	        //addImageToPackage(wordMLPackage, bytes);
+	        
+			wordMLPackage.save(new File(outputDocFinename));
+		} catch (Exception e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		}
+		
+		return outputDocFinename;
+	}
+	
+	private void addImageByImageTitleToPackage(
+			WordprocessingMLPackage wordMLPackage, String title) {
+		AzureSearchResultSet<AzureSearchImageResult> res = imageSearcher.runImageSearch(title);
+		for (AzureSearchImageResult anr : res){
+			String url = anr.getMediaUrl();
+			addImageByURLToPackage( wordMLPackage, url);
+			return;
+		}
+		
+	}
+
+	private void addImageByURLToPackage(WordprocessingMLPackage wordMLPackage,
+            String url){
+		String destinationFile = url.replace("http://", "").replace("/", "_");
+		saveImageFromTheWeb(url, absPath+IMG_REL_PATH+destinationFile);
+		File file = new File(absPath+destinationFile);
+        try {
+			byte[] bytes = convertImageToByteArray(file);
+			addImageToPackage(wordMLPackage, bytes);
+		} catch (FileNotFoundException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		} catch (IOException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		} catch (Exception e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		}
+	}
+	
+	/**
+     *  Docx4j contains a utility method to create an image part from an array of
+     *  bytes and then adds it to the given package. In order to be able to add this
+     *  image to a paragraph, we have to convert it into an inline object. For this
+     *  there is also a method, which takes a filename hint, an alt-text, two ids
+     *  and an indication on whether it should be embedded or linked to.
+     *  One id is for the drawing object non-visual properties of the document, and
+     *  the second id is for the non visual drawing properties of the picture itself.
+     *  Finally we add this inline object to the paragraph and the paragraph to the
+     *  main document of the package.
+     *
+     *  @param wordMLPackage The package we want to add the image to
+     *  @param bytes         The bytes of the image
+     *  @throws Exception    Sadly the createImageInline method throws an Exception
+     *                       (and not a more specific exception type)
+     *                       
+     *                       
+     */
+    protected static void addImageToPackage(WordprocessingMLPackage wordMLPackage,
+                            byte[] bytes) throws Exception {
+        BinaryPartAbstractImage imagePart =
+            BinaryPartAbstractImage.createImagePart(wordMLPackage, bytes);
+ 
+        int docPrId = 1;
+        int cNvPrId = 2;
+            Inline inline = imagePart.createImageInline("Filename hint",
+                "Alternative text", docPrId, cNvPrId, false);
+ 
+        P paragraph = addInlineImageToParagraph(inline);
+ 
+        wordMLPackage.getMainDocumentPart().addObject(paragraph);
+    }
+ 
+    /**
+     *  We create an object factory and use it to create a paragraph and a run.
+     *  Then we add the run to the paragraph. Next we create a drawing and
+     *  add it to the run. Finally we add the inline object to the drawing and
+     *  return the paragraph.
+     *
+     * @param   inline The inline object containing the image.
+     * @return  the paragraph containing the image
+     */
+    private static P addInlineImageToParagraph(Inline inline) {
+        // Now add the in-line image to a paragraph
+    	org.docx4j.wml.ObjectFactory factory = new org.docx4j.wml.ObjectFactory();
+        P paragraph = factory.createP();
+        R run = factory.createR();
+        paragraph.getContent().add(run);
+        Drawing drawing = factory.createDrawing();
+        run.getContent().add(drawing);
+        drawing.getAnchorOrInline().add(inline);
+        return paragraph;
+    }
+    
+    private static CTFootnotes createFootnote(P paragraph){
+    	org.docx4j.wml.ObjectFactory factory = new org.docx4j.wml.ObjectFactory();
+    	CTFootnotes fn = factory.createCTFootnotes();
+    	fn.setParent(paragraph);
+    	
+    	//STFtnEdn sTFtnEdn = factory.createSTFtnEdn();
+    	CTFtnEdn fe = factory.createCTFtnEdn();
+    	fe.setParent(paragraph);
+    	return fn;
+    }
+ 
+    /**
+     * Convert the image from the file into an array of bytes.
+     *
+     * @param file  the image file to be converted
+     * @return      the byte array containing the bytes from the image
+     * @throws FileNotFoundException
+     * @throws IOException
+     */
+    protected static byte[] convertImageToByteArray(File file)
+            throws FileNotFoundException, IOException {
+        InputStream is = new FileInputStream(file );
+        long length = file.length();
+        // You cannot create an array using a long, it needs to be an int.
+        if (length > Integer.MAX_VALUE) {
+            System.out.println("File too large!!");
+        }
+        byte[] bytes = new byte[(int)length];
+        int offset = 0;
+        int numRead = 0;
+        while (offset < bytes.length && (numRead=is.read(bytes, offset, bytes.length-offset)) >= 0) {
+            offset += numRead;
+        }
+        // Ensure all the bytes have been read
+        if (offset < bytes.length) {
+            System.out.println("Could not completely read file "
+                        +file.getName());
+        }
+        is.close();
+        return bytes;
+    }
+    
+    
+    
+    public static void saveImageFromTheWeb(String imageUrl, String destinationFile) {
+		try {
+			URL url = new URL(imageUrl);
+			InputStream is = url.openStream();
+			if (!new File(destinationFile).exists()) {
+				new File(destinationFile).createNewFile();
+			}
+			
+			OutputStream os = new FileOutputStream(destinationFile);
+			
+
+			byte[] b = new byte[2048];
+			int length;
+
+			while ((length = is.read(b)) != -1) {
+				os.write(b, 0, length);
+			}
+
+			is.close();
+			os.close();
+		} catch (MalformedURLException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		} catch (FileNotFoundException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		} catch (IOException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		}
+	}
+    
+    public static void main(String[] args){
+    	WordDocBuilder b = new WordDocBuilder();
+    	List<HitBase> content = new ArrayList<HitBase>();
+    	for(int i = 0; i<10; i++){
+    		HitBase h = new HitBase();
+    		h.setTitle("albert einstein "+i);
+    		List<Fragment> frs = new ArrayList<Fragment>();
+    		frs.add(new Fragment(" content "+i, 0));
+    		h.setFragments(frs);
+    		content.add(h);
+    	}
+    	
+    	b.buildWordDoc(content, "mytitle");
+    }
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/WordDocBuilderEndNotes.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/WordDocBuilderEndNotes.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/WordDocBuilderEndNotes.java
new file mode 100644
index 0000000..7d46f86
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/WordDocBuilderEndNotes.java
@@ -0,0 +1,223 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.similarity.apps.solr;
+
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.math.BigInteger;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+
+import javax.xml.bind.JAXBException;
+
+import net.billylieurance.azuresearch.AzureSearchImageResult;
+import net.billylieurance.azuresearch.AzureSearchResultSet;
+import net.billylieurance.azuresearch.AzureSearchWebResult;
+
+import org.apache.commons.lang.StringUtils;
+import org.docx4j.XmlUtils;
+import org.docx4j.dml.wordprocessingDrawing.Inline;
+import org.docx4j.jaxb.Context;
+import org.docx4j.openpackaging.exceptions.Docx4JException;
+import org.docx4j.openpackaging.exceptions.InvalidFormatException;
+import org.docx4j.openpackaging.packages.WordprocessingMLPackage;
+import org.docx4j.openpackaging.parts.WordprocessingML.BinaryPartAbstractImage;
+import org.docx4j.openpackaging.parts.WordprocessingML.EndnotesPart;
+import org.docx4j.wml.CTEndnotes;
+import org.docx4j.wml.CTFtnEdn;
+import org.docx4j.wml.Drawing;
+import org.docx4j.wml.P;
+import org.docx4j.wml.R;
+
+import opennlp.tools.similarity.apps.BingQueryRunner;
+import opennlp.tools.similarity.apps.Fragment;
+import opennlp.tools.similarity.apps.HitBase;
+
+public class WordDocBuilderEndNotes extends WordDocBuilderSingleImageSearchCall{
+	
+	public String buildWordDoc(List<HitBase> content, String title){
+		
+		String outputDocFinename =  absPath+"written/"+ title.replace(' ','_').replace('\"', ' ').trim()+ ".docx";
+		
+		WordprocessingMLPackage wordMLPackage=null;
+		
+       
+		List<String> imageURLs = getAllImageSearchResults(title);
+		int count=0;
+		BigInteger refId = BigInteger.ONE;
+		try {
+			wordMLPackage = WordprocessingMLPackage.createPackage();
+			
+			
+			CTEndnotes endnotes = null;
+			try {
+				EndnotesPart ep = new EndnotesPart();
+				endnotes = Context.getWmlObjectFactory().createCTEndnotes();
+				ep.setJaxbElement(endnotes);
+				wordMLPackage.getMainDocumentPart().addTargetPart(ep);
+			} catch (InvalidFormatException e1) {
+				// TODO Auto-generated catch block
+				e1.printStackTrace();
+			}
+			
+			
+			
+			
+			wordMLPackage.getMainDocumentPart().addStyledParagraphOfText("Title", title.toUpperCase());
+			for(HitBase para: content){
+				if (para.getFragments()==null || para.getFragments().size()<1) // no found content in this hit
+						continue;
+				try {
+					String processedParaTitle = processParagraphTitle(para.getTitle());
+					
+					if (processedParaTitle!=null && 
+							!processedParaTitle.endsWith("..") || StringUtils.isAlphanumeric(processedParaTitle)){
+						wordMLPackage.getMainDocumentPart().addStyledParagraphOfText("Subtitle",processedParaTitle);
+					}
+					String paraText = processParagraphText(para.getFragments().toString());
+					wordMLPackage.getMainDocumentPart().addParagraphOfText(paraText);
+					
+					 CTFtnEdn endnote = Context.getWmlObjectFactory().createCTFtnEdn();
+			         endnotes.getEndnote().add(endnote);
+			        
+			         endnote.setId(refId);
+			         refId.add(BigInteger.ONE);
+			         String url = para.getUrl();
+			         String endnoteBody = "<w:p xmlns:w=\"http://schemas.openxmlformats.org/wordprocessingml/2006/main\" ><w:pPr><w:pStyle w:val=\"EndnoteText\"/></w:pPr><w:r><w:rPr>" +
+			         		"<w:rStyle w:val=\"EndnoteReference\"/></w:rPr><w:endnoteRef/></w:r><w:r><w:t xml:space=\"preserve\"> "+ url + "</w:t></w:r></w:p>";
+			         try {
+						endnote.getEGBlockLevelElts().add( XmlUtils.unmarshalString(endnoteBody));
+					} catch (JAXBException e) {
+						// TODO Auto-generated catch block
+						e.printStackTrace();
+					}
+			         
+			         // Add the body text referencing it
+			         String docBody = "<w:p xmlns:w=\"http://schemas.openxmlformats.org/wordprocessingml/2006/main\" ><w:r><w:t>"//+ paraText
+			         /*+ refId.toString()*/ +"</w:t></w:r><w:r><w:rPr><w:rStyle w:val=\"EndnoteReference\"/></w:rPr><w:endnoteReference w:id=\""+refId.toString()+"\"/></w:r></w:p>";
+			         
+			         try {
+			        	 wordMLPackage.getMainDocumentPart().addParagraph(docBody);
+					} catch (JAXBException e) {
+						// TODO Auto-generated catch block
+						e.printStackTrace();
+					}
+					
+					try {
+						addImageByImageURLToPackage(count, wordMLPackage, imageURLs);
+					} catch (Exception e) {
+						// no need to report issues
+						//e.printStackTrace();
+					}
+				} catch (Exception e) {
+					// TODO Auto-generated catch block
+					e.printStackTrace();
+				}
+				count++;
+			}
+			// now add URLs
+			wordMLPackage.getMainDocumentPart().addStyledParagraphOfText("Subtitle", "REFERENCES");
+			for(HitBase para: content){
+				if (para.getFragments()==null || para.getFragments().size()<1) // no found content in this hit
+						continue;
+				try {
+					wordMLPackage.getMainDocumentPart().addStyledParagraphOfText("Subtitle",
+							para.getTitle());
+					String paraText = para.getUrl();
+					wordMLPackage.getMainDocumentPart().addParagraphOfText(paraText);
+					
+					
+				} catch (Exception e) {
+					// TODO Auto-generated catch block
+					e.printStackTrace();
+				}
+			}
+	
+	        
+			try {
+				wordMLPackage.save(new File(outputDocFinename));
+				System.out.println("Finished creating docx ="+outputDocFinename);
+			} catch (Exception e) {
+				// TODO Auto-generated catch block
+				e.printStackTrace();
+			}
+			
+			try {
+				String fileNameToDownload = "/var/www/wrt_latest/"+title.replace(' ','_').replace('\"', ' ').trim()+ ".docx";
+				wordMLPackage.save(new File(fileNameToDownload));
+				System.out.println("Wrote a doc for download :"+fileNameToDownload);
+			} catch (Exception e) {
+				// TODO Auto-generated catch block
+				e.printStackTrace();
+			}
+			
+		} catch (Exception e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		}
+		
+		return outputDocFinename;
+	}
+	
+	public static String processParagraphText(String title){
+		
+		return title.replace("[", "").replace("]", "").replace(" | ", "")
+		.replace(".,", ".").replace(".\"", "\"").replace(". .", ".")
+		.replace(",.", ".");
+	}
+	
+	public static String processParagraphTitle(String title){
+		String titleDelim = title.replace('-', '&').replace('|', '&');
+		String[] titleParts = titleDelim.split("&");
+		
+		int lenCurr = -1; 
+		String bestPart = null;
+		for(String candidatePart: titleParts ){ // if this part longer and does not have periods
+			if (lenCurr< candidatePart.length() && candidatePart.indexOf('.')<0){
+				lenCurr = candidatePart.length();
+				bestPart = candidatePart;
+			}
+		}
+		
+		return bestPart;
+	}
+
+    
+    public static void main(String[] args){
+    	WordDocBuilderEndNotes b = new WordDocBuilderEndNotes();
+    	List<HitBase> content = new ArrayList<HitBase>();
+    	for(int i = 0; i<10; i++){
+    		HitBase h = new HitBase();
+    		h.setTitle("albert einstein "+i);
+    		List<Fragment> frs = new ArrayList<Fragment>();
+    		frs.add(new Fragment(" content "+i, 0));
+    		h.setFragments(frs);
+    		h.setUrl("http://www."+i+".com");
+    		content.add(h);
+    	}
+    	
+    	b.buildWordDoc(content, "albert einstein");
+    }
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/WordDocBuilderSingleImageSearchCall.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/WordDocBuilderSingleImageSearchCall.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/WordDocBuilderSingleImageSearchCall.java
new file mode 100644
index 0000000..7c7c1d9
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/solr/WordDocBuilderSingleImageSearchCall.java
@@ -0,0 +1,174 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.similarity.apps.solr;
+
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.ArrayList;
+import java.util.List;
+
+import net.billylieurance.azuresearch.AzureSearchImageResult;
+import net.billylieurance.azuresearch.AzureSearchResultSet;
+import net.billylieurance.azuresearch.AzureSearchWebResult;
+
+import org.apache.commons.lang.StringUtils;
+//import org.docx4j.Docx4J;
+//import org.docx4j.convert.out.FOSettings;
+import org.docx4j.openpackaging.packages.WordprocessingMLPackage;
+
+import opennlp.tools.similarity.apps.ContentGeneratorSupport;
+import opennlp.tools.similarity.apps.Fragment;
+import opennlp.tools.similarity.apps.HitBase;
+
+
+
+public class WordDocBuilderSingleImageSearchCall extends WordDocBuilder{
+	
+	public String buildWordDoc(List<HitBase> content, String title){
+		
+		String outputDocFinename =  absPath+"/written/"+ title.replace(' ','_').replace('\"', ' ').trim()+ ".docx";
+		
+		WordprocessingMLPackage wordMLPackage;
+		List<String> imageURLs = getAllImageSearchResults(title);
+		int count=0;
+		try {
+			wordMLPackage = WordprocessingMLPackage.createPackage();
+			wordMLPackage.getMainDocumentPart().addStyledParagraphOfText("Title", title.toUpperCase());
+			for(HitBase para: content){
+				if (para.getFragments()==null || para.getFragments().size()<1) // no found content in this hit
+						continue;
+				try {
+					if (!para.getTitle().endsWith("..") /*|| StringUtils.isAlphanumeric(para.getTitle())*/){
+						String sectTitle = ContentGeneratorSupport.getPortionOfTitleWithoutDelimiters(para.getTitle());
+						wordMLPackage.getMainDocumentPart().addStyledParagraphOfText("Subtitle",
+							sectTitle);
+					}
+					String paraText = para.getFragments().toString().replace("[", "").replace("]", "").replace(" | ", "")
+							.replace(".,", ".").replace(".\"", "\"").replace(". .", ".")
+							.replace(",.", ".");
+					wordMLPackage.getMainDocumentPart().addParagraphOfText(paraText);
+					
+					try {
+						addImageByImageURLToPackage(count, wordMLPackage, imageURLs);
+					} catch (Exception e) {
+						// TODO Auto-generated catch block
+						e.printStackTrace();
+					}
+				} catch (Exception e) {
+					// TODO Auto-generated catch block
+					e.printStackTrace();
+				}
+				count++;
+			}
+			// now add URLs
+			wordMLPackage.getMainDocumentPart().addStyledParagraphOfText("Subtitle", "REFERENCES");
+			for(HitBase para: content){
+				if (para.getFragments()==null || para.getFragments().size()<1) // no found content in this hit
+						continue;
+				try {
+					wordMLPackage.getMainDocumentPart().addStyledParagraphOfText("Subtitle",
+							para.getTitle());
+					String paraText = para.getUrl();
+					wordMLPackage.getMainDocumentPart().addParagraphOfText(paraText);
+					
+					
+				} catch (Exception e) {
+					// TODO Auto-generated catch block
+					e.printStackTrace();
+				}
+			}
+	
+	        
+			wordMLPackage.save(new File(outputDocFinename));
+			System.out.println("Finished creating docx ="+outputDocFinename);
+		//TODO pdf export
+			/*
+			FOSettings foSettings = Docx4J.createFOSettings();
+            foSettings.setWmlPackage(wordMLPackage);
+            OutputStream os = new java.io.FileOutputStream(outputDocFinename.replace(".docx", ".pdf"));
+            Docx4J.toFO(foSettings, os, Docx4J.FLAG_NONE);
+        	System.out.println("Finished creating docx's PDF ="+outputDocFinename);
+    	*/	
+			
+		} catch (Exception e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		}
+		
+		return outputDocFinename;
+	}
+	
+	protected void addImageByImageURLToPackage(int count,
+			WordprocessingMLPackage wordMLPackage,
+			List<String>  imageURLs) {
+		if (count>imageURLs.size()-1)
+			return;
+		
+		String url = imageURLs.get(count);
+		String destinationFile = url.replace("http://", "").replace("/", "_");
+		saveImageFromTheWeb(url, absPath+IMG_REL_PATH+destinationFile);
+		File file = new File(absPath+IMG_REL_PATH+destinationFile);
+        try {
+			byte[] bytes = convertImageToByteArray(file);
+			addImageToPackage(wordMLPackage, bytes);
+		} catch (FileNotFoundException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		} catch (IOException e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		} catch (Exception e) {
+			// TODO Auto-generated catch block
+			e.printStackTrace();
+		}
+		
+	}
+
+	protected List<String>  getAllImageSearchResults(String title) {
+		List<String> imageURLs = new ArrayList<String>();
+		AzureSearchResultSet<AzureSearchImageResult> res = imageSearcher.runImageSearch(title);
+		for(AzureSearchImageResult imResult: res){
+			imageURLs.add(imResult.getMediaUrl());
+		}
+		return imageURLs;
+		
+	}
+
+    
+    public static void main(String[] args){
+    	WordDocBuilderSingleImageSearchCall b = new WordDocBuilderSingleImageSearchCall();
+    	List<HitBase> content = new ArrayList<HitBase>();
+    	for(int i = 0; i<10; i++){
+    		HitBase h = new HitBase();
+    		h.setTitle("albert einstein "+i);
+    		List<Fragment> frs = new ArrayList<Fragment>();
+    		frs.add(new Fragment(" content "+i, 0));
+    		h.setFragments(frs);
+    		content.add(h);
+    	}
+    	
+    	b.buildWordDoc(content, "albert einstein");
+    }
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/taxo_builder/CsvAdapter.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/taxo_builder/CsvAdapter.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/taxo_builder/CsvAdapter.java
new file mode 100644
index 0000000..8d718f3
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/taxo_builder/CsvAdapter.java
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.similarity.apps.taxo_builder;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import opennlp.tools.jsmlearning.ProfileReaderWriter;
+
+public class CsvAdapter {
+	  Map<String, List<List<String>>> lemma_AssocWords = new HashMap<String, List<List<String>>>();
+	private String resourceDir=null, fileNameToImport = null;
+	  
+	  public CsvAdapter(){
+		  if (resourceDir==null)
+				try {
+					resourceDir = new File( "." ).getCanonicalPath()+"/src/test/resources";
+				} catch (IOException e) {
+					e.printStackTrace();
+				}
+		  fileNameToImport = resourceDir + "/taxonomies/musicTaxonomyRoot.csv";
+	  }
+	  
+	  public void importCSV(){
+		  List<String[]> lines = ProfileReaderWriter.readProfiles(fileNameToImport);
+		  String topNode=null;
+		  for(String[] line: lines){	
+			  String line0 = extractEntity(line[0]).toLowerCase();
+			  List<String> path = new ArrayList<String>();
+			  List<List<String>> paths = new ArrayList<List<String>>();
+			if (line[1]!=null && line[1].equals("1")){
+				  topNode = line0;
+			} else {
+				path.add(topNode);
+				path.add(line0);
+				paths.add(path);
+				lemma_AssocWords.put(line0, paths);			
+			}
+		  }
+	  }
+
+	private String extractEntity(String s) {
+		Integer[] poss = new Integer[]{s.indexOf('/'),
+				s.indexOf('('),	 s.indexOf('_')};
+		
+		int cutPos = 100;
+		for(int p: poss){
+			if (p>-1 && p< cutPos)
+				cutPos=p;
+		}
+		
+		if (cutPos<100)
+			s = s.substring(0,cutPos).trim();
+		return s;
+	}
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/taxo_builder/DomainTaxonomyExtender.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/taxo_builder/DomainTaxonomyExtender.java b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/taxo_builder/DomainTaxonomyExtender.java
new file mode 100644
index 0000000..8538c25
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/similarity/apps/taxo_builder/DomainTaxonomyExtender.java
@@ -0,0 +1,230 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.similarity.apps.taxo_builder;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.logging.Logger;
+
+import org.apache.commons.lang.StringUtils;
+
+import opennlp.tools.parse_thicket.matching.Matcher;
+import opennlp.tools.similarity.apps.BingQueryRunner;
+import opennlp.tools.similarity.apps.HitBase;
+import opennlp.tools.similarity.apps.utils.StringCleaner;
+import opennlp.tools.stemmer.PStemmer;
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+import opennlp.tools.textsimilarity.ParseTreeChunkListScorer;
+import opennlp.tools.textsimilarity.SentencePairMatchResult;
+import opennlp.tools.textsimilarity.TextProcessor;
+import opennlp.tools.textsimilarity.chunker2matcher.ParserChunker2MatcherProcessor;
+
+/**
+
+ * 
+ */
+
+public class DomainTaxonomyExtender {
+	private static Logger LOG = Logger
+			.getLogger("opennlp.tools.similarity.apps.taxo_builder.DomainTaxonomyExtender");
+
+	private BingQueryRunner brunner = new BingQueryRunner();
+
+	protected static String BING_KEY = "WFoNMM706MMJ5JYfcHaSEDP+faHj3xAxt28CPljUAHA";
+	Matcher matcher = new Matcher(); 
+
+	private final static String TAXO_FILENAME = "taxo_data.dat";
+
+	private Map<String, List<List<String>>> lemma_ExtendedAssocWords = new HashMap<String, List<List<String>>>();
+	private Map<List<String>, List<List<String>>> assocWords_ExtendedAssocWords = new HashMap<List<String>, List<List<String>>>();
+	private PStemmer ps;
+
+	CsvAdapter adapter = new CsvAdapter();
+
+	public Map<List<String>, List<List<String>>> getAssocWords_ExtendedAssocWords() {
+		return assocWords_ExtendedAssocWords;
+	}
+
+	public Map<String, List<List<String>>> getLemma_ExtendedAssocWords() {
+		return lemma_ExtendedAssocWords;
+	}
+
+	public void setLemma_ExtendedAssocWords(
+			Map<String, List<List<String>>> lemma_ExtendedAssocWords) {
+		this.lemma_ExtendedAssocWords = lemma_ExtendedAssocWords;
+	}
+
+	public DomainTaxonomyExtender() {
+		ps = new PStemmer();
+		adapter.importCSV();
+		brunner.setKey(BING_KEY);
+	}
+
+	private List<List<String>> getCommonWordsFromList_List_ParseTreeChunk(
+			List<List<ParseTreeChunk>> matchList, List<String> queryWordsToRemove,
+			List<String> toAddAtEnd) {
+		List<List<String>> res = new ArrayList<List<String>>();
+		for (List<ParseTreeChunk> chunks : matchList) {
+			List<String> wordRes = new ArrayList<String>();
+			for (ParseTreeChunk ch : chunks) {
+				List<String> lemmas = ch.getLemmas();
+				for (int w = 0; w < lemmas.size(); w++)
+					if ((!lemmas.get(w).equals("*"))
+							&& ((ch.getPOSs().get(w).startsWith("NN") || ch.getPOSs().get(w).startsWith("JJ") || ch.getPOSs().get(w)
+									.startsWith("VB"))) && lemmas.get(w).length() > 2) {
+						String formedWord = lemmas.get(w);
+						String stemmedFormedWord = ps.stem(formedWord);
+						if (!stemmedFormedWord.startsWith("invalid"))
+							wordRes.add(formedWord);
+					}
+			}
+			wordRes = new ArrayList<String>(new HashSet<String>(wordRes));
+			List<String> cleanedRes = new ArrayList<String>();
+			for(String s: wordRes){
+				if (!queryWordsToRemove.contains(s))
+					cleanedRes .add(s);  	  
+			}
+			//wordRes.removeAll(queryWordsToRemove);
+			if (cleanedRes.size() > 0) {
+				//cleanedRes.addAll(toAddAtEnd);
+				res.add(cleanedRes);
+			}
+		}
+		res = new ArrayList<List<String>>(new HashSet<List<String>>(res));
+		return res;
+	}
+
+	public void extendTaxonomy(String fileNameWithTaxonomyRoot, String domain, String lang) {
+
+
+		List<String> entries = new ArrayList<String>((adapter.lemma_AssocWords.keySet()));
+		try {
+			for (String entity : entries) { // .
+				List<List<String>> paths = adapter.lemma_AssocWords.get(entity);
+				for (List<String> taxoPath : paths) {
+					String query = taxoPath.toString() + " " + entity + " " + domain; 
+					query = query.replace('[', ' ').replace(']', ' ').replace(',', ' ')
+							.replace('_', ' ');
+					List<List<ParseTreeChunk>> matchList = runSearchForTaxonomyPath(
+							query, "", lang, 20); //30
+					List<String> toRemoveFromExtension = new ArrayList<String>(taxoPath);
+					toRemoveFromExtension.add(entity);
+					toRemoveFromExtension.add(domain);
+					List<List<String>> resList = getCommonWordsFromList_List_ParseTreeChunk(
+							matchList, toRemoveFromExtension, taxoPath);
+					assocWords_ExtendedAssocWords.put(taxoPath, resList);
+					resList.add(taxoPath);
+					lemma_ExtendedAssocWords.put(entity, resList);
+
+					TaxonomySerializer ser = new TaxonomySerializer(lemma_ExtendedAssocWords,
+							assocWords_ExtendedAssocWords);
+					ser.writeTaxonomy(TAXO_FILENAME);
+				}
+			}
+		} catch (Exception e) {
+			e.printStackTrace();
+			System.err.println("Problem taxonomy matching");
+		}
+
+
+	}
+
+	public List<List<ParseTreeChunk>> runSearchForTaxonomyPath(String query,
+			String domain, String lang, int numbOfHits) {
+		List<List<ParseTreeChunk>> genResult = new ArrayList<List<ParseTreeChunk>>();
+		try {
+			List<HitBase> resultList = brunner.runSearch(query, numbOfHits);
+
+			for (int i = 0; i < resultList.size(); i++) {
+				{
+					for (int j = i + 1; j < resultList.size(); j++) {
+						HitBase h1 = resultList.get(i);
+						HitBase h2 = resultList.get(j);
+						String snapshot1 = StringCleaner.processSnapshotForMatching(h1
+								.getTitle() + " " + h1.getAbstractText());
+						String snapshot2 = StringCleaner.processSnapshotForMatching(h2
+								.getTitle() + " " + h2.getAbstractText());
+						List<List<ParseTreeChunk>> overlaps =matcher.assessRelevance(snapshot1, snapshot2);
+						genResult.addAll(overlaps);
+					}
+				}
+			}
+
+		} catch (Exception e) {
+			e.printStackTrace();
+			System.err.print("Problem searching for "+query);
+		}
+
+		return genResult;
+	}
+
+	public List<String> runSearchForTaxonomyPathFlatten(String query,
+			String domain, String lang, int numbOfHits) {
+		List<String> genResult = new ArrayList<String>();
+		try {
+			List<HitBase> resultList = brunner.runSearch(query, numbOfHits);
+
+			for (int i = 0; i < resultList.size(); i++) {
+				{
+					for (int j = i + 1; j < resultList.size(); j++) {
+						HitBase h1 = resultList.get(i);
+						HitBase h2 = resultList.get(j);
+						String snapshot1 = StringCleaner.processSnapshotForMatching(h1
+								.getTitle() + " " + h1.getAbstractText());
+						String snapshot2 = StringCleaner.processSnapshotForMatching(h2
+								.getTitle() + " " + h2.getAbstractText());
+						List<String> overlaps =assessKeywordOverlap(snapshot1, snapshot2);
+						genResult.addAll(overlaps);
+					}
+				}
+			}
+
+		} catch (Exception e) {
+			System.err.print("Problem searching for "+query);
+		}
+
+		return genResult;
+	}
+
+
+
+	private List<String> assessKeywordOverlap(String snapshot1, String snapshot2) {
+		List<String> results = new ArrayList<String>();
+		List<String> firstList = TextProcessor.fastTokenize(snapshot1, false), 
+				secondList = TextProcessor.fastTokenize(snapshot2, false);	  
+		firstList.retainAll(secondList);
+		for(String s: firstList){
+			if (s.length()<4)
+				continue;
+			if (!StringUtils.isAlpha(s))
+				continue;
+			results.add(s);
+		}
+		return results;
+	}
+
+	public static void main(String[] args) {
+		DomainTaxonomyExtender self = new DomainTaxonomyExtender();
+		self.extendTaxonomy("", "music",
+				"en");
+
+	}
+
+}


[49/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/fca/BasicLevelMetrics.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/fca/BasicLevelMetrics.java b/opennlp-similarity/src/main/java/opennlp/tools/fca/BasicLevelMetrics.java
new file mode 100755
index 0000000..f1f6bd0
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/fca/BasicLevelMetrics.java
@@ -0,0 +1,676 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.fca;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.commons.collections.ListUtils;
+
+import edu.stanford.nlp.io.EncodingPrintWriter.out;
+
+public class BasicLevelMetrics {
+
+	
+	ConceptLattice cl;	
+	ArrayList<ArrayList<Integer>> attributesExtent  = null; 
+	ArrayList<ArrayList<Integer>> objectsIntent  = null; 
+	ArrayList<Integer> attributes = null; 
+	double[][] objectsSimilarityJ = null;
+	double [][] objectsSimilaritySMC = null; 
+					
+	
+	public BasicLevelMetrics (ConceptLattice cl){
+		this.cl = cl;
+		this.attributesExtent = null;
+		objectsSimilarityJ = new double [cl.objectCount][cl.objectCount];
+		objectsSimilaritySMC = new double [cl.objectCount][cl.objectCount];
+	}
+	
+	public void setUp(){
+		attributesExtent = new ArrayList<ArrayList<Integer>>();
+		objectsIntent = new ArrayList<ArrayList<Integer>>();
+		attributes = new ArrayList<Integer>();
+		
+		for (int i=0;i<cl.attributeCount;i++){
+			attributesExtent.add((ArrayList<Integer>) cl.getAttributeExtByID(i));
+			attributes.add(i);
+		}	
+		
+		for (int i=0;i<cl.objectCount;i++){
+			objectsIntent.add((ArrayList<Integer>) cl.getObjectIntByID(i));
+		}
+	
+		double [] buf = new double[2];
+		
+		for (int i = 0; i < cl.objectCount; i++){
+			for (int j = i + 1 ; j < cl.objectCount; j++){
+					buf = simJ_SMC(objectsIntent.get(i), objectsIntent.get(j));	
+					objectsSimilarityJ[i][j] = buf[0];
+					objectsSimilarityJ[j][i] = buf[0];
+					objectsSimilaritySMC[i][j] = buf[1];
+					objectsSimilaritySMC[j][i] = buf[1];
+			}
+			objectsSimilarityJ[i][i] = 1;
+			objectsSimilaritySMC[i][i] = 1;
+		}
+		
+		//System.out.println("J");
+		//System.out.println(Arrays.deepToString(objectsSimilarityJ));
+		//System.out.println("SMC");
+		//System.out.println(Arrays.deepToString(objectsSimilaritySMC));
+
+	} 
+	 
+	//Utility functions for  Similarity approach (S)
+	public double simSMC (ArrayList<Integer> intent1, ArrayList<Integer>intent2){
+		int tp = (ListUtils.intersection(intent1,intent2)).size();
+		ArrayList<Integer> fnlst = new ArrayList<Integer>();
+		fnlst.addAll(this.attributes);
+		fnlst.removeAll(ListUtils.union(intent1,intent2)); 
+		int fn = fnlst.size();
+		return (this.attributes.size()>0) ? 1.*(tp + fn)/this.attributes.size() : 0;
+	}
+	
+	public double simJ (ArrayList<Integer> intent1, ArrayList<Integer>intent2){
+		return 1.*(ListUtils.intersection(intent1,intent2)).size()/(ListUtils.union(intent1,intent2)).size() ;
+	}
+	
+	public  double [] simJ_SMC(ArrayList<Integer> intent1, ArrayList<Integer>intent2){
+		double simJ = 0;
+		double simSMC = 0;	
+		Set<Integer> intersection = new HashSet<Integer>(); 
+		intersection.addAll(intent1);
+		intersection.retainAll(intent2);
+		
+		Set<Integer> union = new HashSet<Integer>(); 
+		union.addAll(intent1);
+		union.addAll(intent2);
+		int fn = 0;
+		Set<Integer> unionOut = new HashSet<Integer>();
+		unionOut.addAll(this.attributes);
+		unionOut.removeAll(union);	
+		simSMC = (this.attributes.size() > 0) ? 1.*(intersection.size() + unionOut.size())/this.attributes.size() : 0;
+		simJ = (union.size() > 0) ? 1.*intersection.size()/union.size() : 0;
+	    return new double[] {simJ, simSMC};
+	}
+	
+	
+	
+	public double avgCohSMC (FormalConcept c){
+		double sum = 0;
+		if (c.extent.size() == 1)
+			return 1.;//c.intent.size();
+		if (c.extent.size() == 0)
+			return 0.;
+		else {
+			for (Integer i:c.extent){
+				for (Integer j: c.extent){
+					if (i<j)
+						sum+=objectsSimilaritySMC[i][j];
+				}				
+			}
+			return (c.extent.size() > 1 ) ? 2.*sum/c.extent.size()/(c.extent.size()-1) : 0;// \ufffd\ufffd 2, \ufffd\ufffd\ufffd \ufffd\ufffd\ufffd \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd
+		}
+	}
+	
+	public double avgCohJ (FormalConcept c){
+		double sum = 0;
+		if (c.extent.size() == 1)
+			return 1.;//c.intent.size();
+		if (c.extent.size() == 0)
+			return 0.;
+		else {
+			for (Integer i:c.extent){
+				for (Integer j: c.extent){
+					if (i<j){
+						sum+=objectsSimilarityJ[i][j];
+					}
+				}				
+			}
+			return (c.extent.size() > 1 ) ? 2.*sum/c.extent.size()/(c.extent.size()-1) : 0; 
+		}
+	}
+	
+	public double minCohJ (FormalConcept c){
+		double min = Integer.MAX_VALUE,
+				val = 0;
+		
+		for (Integer i:c.extent){
+			for (Integer j: c.extent){
+					val = objectsSimilarityJ[i][j];
+					if (val<min)
+						min = val;
+			}				
+		}
+		return (min < Integer.MAX_VALUE) ? min : 0;
+	}
+	
+	public double minCohSMC (FormalConcept c){
+		double min = Integer.MAX_VALUE,
+				val = 0;
+		for (Integer i:c.extent){
+			for (Integer j: c.extent){
+					val = objectsSimilaritySMC[i][j];
+					if (val<min)
+						min = val;
+			}				
+		}
+		return (min < Integer.MAX_VALUE) ? min : 0;
+	}
+
+	
+	public double upperCohAvgByAvgJ(FormalConcept c, float tetta){
+		//average alpha with average cohesion J
+		double sum = 0;
+		Set<Integer> upperNeighbors =c.parents;
+		int rightNeighborsNumber = 0;
+		float truthDegree = 0;
+		for (Integer i: upperNeighbors){
+			if (c.cohAvgJ > cl.conceptList.get(i).cohAvgJ){
+				rightNeighborsNumber++;
+				sum+=1.*cl.conceptList.get(i).cohAvgJ/c.cohAvgJ;	
+			}
+		}
+		if (rightNeighborsNumber == 0)
+			return 0;
+		else{
+			truthDegree = (1.*rightNeighborsNumber/upperNeighbors.size()>=tetta) ? 1 : 0;
+			//truthDegree =  (float) Math.min(1.0, truthDegree);
+			return (1-sum/rightNeighborsNumber)*truthDegree;
+		}
+	}
+	
+	public double upperCohAvgByMinJ(FormalConcept c,float tetta){
+		//average alpha with min cohesion J
+		double sum = 0;
+		Set<Integer> upperNeighbors =c.parents;
+		int rightNeighborsNumber = 0;
+		float truthDegree = 0;
+		for (Integer i: upperNeighbors){
+			if (c.cohMinJ > cl.conceptList.get(i).cohMinJ){
+				rightNeighborsNumber++;
+				sum+=1.*cl.conceptList.get(i).cohMinJ/c.cohMinJ;	
+			}
+		}
+		if (rightNeighborsNumber == 0)
+			return 0;
+		else{
+			truthDegree = (1.*rightNeighborsNumber/upperNeighbors.size()>=tetta) ? 1 : 0;
+			truthDegree =  (float) Math.min(1.0, truthDegree);
+			return (1-sum/rightNeighborsNumber)*truthDegree;
+		}
+	}
+	
+	public double upperCohMinByAvgJ(FormalConcept c,float tetta){
+		//min alpha whth average cohesion J
+		double max = Integer.MIN_VALUE,
+				val = 0; 
+		int rightNeighborsNumber = 0;
+		float truthDegree = 0;
+		Set<Integer> upperNeighbors =c.parents;
+		for (Integer i: upperNeighbors){
+			if (c.cohAvgJ > cl.conceptList.get(i).cohAvgJ){
+				rightNeighborsNumber++;
+				val = 1.*cl.conceptList.get(i).cohAvgJ/c.cohAvgJ;	
+				if (val>max)
+					max = val;
+			}
+		}
+		if (rightNeighborsNumber == 0)
+			return 0;
+		else{
+			truthDegree = (1.*rightNeighborsNumber/upperNeighbors.size()>=tetta) ? 1 : 0;
+			truthDegree =  (float) Math.min(1.0, truthDegree);
+			return (1-max)*truthDegree;
+		}		
+	}
+	
+	public double upperCohMinByMinJ(FormalConcept c,float tetta){
+		//min alpha whth average cohesion J
+		double max = Integer.MIN_VALUE,
+				val = 0; 
+		int rightNeighborsNumber = 0;
+		float truthDegree = 0;
+		Set<Integer> upperNeighbors =c.parents;
+		for (Integer i: upperNeighbors){
+			if (c.cohMinJ > cl.conceptList.get(i).cohMinJ){
+				rightNeighborsNumber++;
+				val = 1.*cl.conceptList.get(i).cohMinJ/c.cohMinJ;	
+				if (val>max)
+					max = val;
+			}
+		}
+		if (rightNeighborsNumber == 0)
+			return 0;
+		else{
+			//truthDegree = rightNeighborsNumber/tetta/upperNeighbors.size();
+			truthDegree = (1.*rightNeighborsNumber/upperNeighbors.size()>=tetta) ? 1 : 0;
+			truthDegree =  (float) Math.min(1.0, truthDegree);			
+			return (1-max)*truthDegree;
+		}		
+	}
+	
+	public double upperCohAvgByAvgSMC(FormalConcept c,float tetta){
+		//average alpha with average cohesion SMC
+		double sum = 0;
+		Set<Integer> upperNeighbors =c.parents;
+		int rightNeighborsNumber = 0;
+		float truthDegree = 0;
+		for (Integer i: upperNeighbors){
+			if (c.cohAvgSMC > cl.conceptList.get(i).cohAvgSMC){
+				rightNeighborsNumber++;
+				sum+=1.*cl.conceptList.get(i).cohAvgSMC/c.cohAvgSMC;		
+			}
+		}
+		if (rightNeighborsNumber == 0)
+			return 0;
+		else{
+			truthDegree = (1.*rightNeighborsNumber/upperNeighbors.size()>=tetta) ? 1 : 0;
+			return (1-sum/rightNeighborsNumber)*truthDegree;
+		}	
+	}
+	
+	public double upperCohAvgByMinSMC(FormalConcept c,float tetta){
+		//average alpha with min cohesion SMC
+		double sum = 0;
+		Set<Integer> upperNeighbors =c.parents;
+		int rightNeighborsNumber = 0;
+		float truthDegree = 0;
+		for (Integer i: upperNeighbors){
+			if (c.cohMinSMC > cl.conceptList.get(i).cohMinSMC){
+				rightNeighborsNumber++;
+				sum+=1.*cl.conceptList.get(i).cohMinSMC/c.cohMinSMC;		
+			}
+		}
+		if (rightNeighborsNumber == 0)
+			return 0;
+		else{
+			truthDegree = (1.*rightNeighborsNumber/upperNeighbors.size()>=tetta) ? 1 : 0;
+			return (1-sum/rightNeighborsNumber)*truthDegree;
+		}
+	}
+		
+		
+	public double upperCohMinByAvgSMC(FormalConcept c,float tetta){
+		//min alpha whth average cohesion J
+		
+		double max = Integer.MIN_VALUE,
+				val = 0; 
+		int rightNeighborsNumber = 0;
+		float truthDegree = 0;
+		Set<Integer> upperNeighbors =c.parents;
+		for (Integer i: upperNeighbors){
+			if (c.cohAvgSMC > cl.conceptList.get(i).cohAvgSMC){
+				rightNeighborsNumber++;
+				val = 1.*cl.conceptList.get(i).cohAvgSMC/c.cohAvgSMC;	
+				if (val>max)
+					max = val;
+			}
+		}
+		if (rightNeighborsNumber == 0)
+			return 0;
+		else{
+			truthDegree = (1.*rightNeighborsNumber/upperNeighbors.size()>=tetta) ? 1 : 0;
+			truthDegree =  (float) Math.min(1.0, truthDegree);
+			return (1-max)*truthDegree;
+		}			
+	}
+	
+	public double upperCohMinByMinSMC(FormalConcept c,float tetta){
+		//min alpha whth average cohesion J
+		double max = Integer.MIN_VALUE,
+				val = 0; 
+		int rightNeighborsNumber = 0;
+		float truthDegree = 0;
+		Set<Integer> upperNeighbors =c.parents;
+		for (Integer i: upperNeighbors){
+			if (c.cohMinSMC > cl.conceptList.get(i).cohMinSMC){
+				rightNeighborsNumber++;
+				val = 1.*cl.conceptList.get(i).cohMinSMC/c.cohMinSMC;	
+				if (val>max)
+					max = val;
+			}
+		}
+		if (rightNeighborsNumber == 0)
+			return 0;
+		else{
+			truthDegree = (1.*rightNeighborsNumber/upperNeighbors.size()>=tetta) ? 1 : 0;
+			truthDegree =  (float) Math.min(1.0, truthDegree);		
+			return (1-max)*truthDegree;
+		}		
+	}
+	
+
+	
+	public double lowerCohAvgByAvgJ(FormalConcept c,float tetta){
+		double sum = 0;
+		Set<Integer> lowerNeighbors =c.childs;
+		int rightNeighborsNumber = 0;
+		float truthDegree = 0;
+		for (Integer i: lowerNeighbors){
+			if (c.cohAvgJ < cl.conceptList.get(i).cohAvgJ){
+				rightNeighborsNumber++;
+				sum+=1.*c.cohAvgJ/cl.conceptList.get(i).cohAvgJ;
+			}
+		}
+		if (rightNeighborsNumber == 0)
+			return 0;
+		else{
+			truthDegree = (1.*rightNeighborsNumber/lowerNeighbors.size()>=tetta) ? 1 : 0;
+			truthDegree =  (float) Math.min(1.0, truthDegree);
+			return (sum/rightNeighborsNumber)*truthDegree;
+		}
+	}
+	
+	public double lowerCohAvgByMinJ(FormalConcept c,float tetta){
+		double sum = 0;
+		int rightNeighborsNumber = 0;
+		float truthDegree = 0;
+		Set<Integer> lowerNeighbors =c.childs;
+		for (Integer i: lowerNeighbors){
+			if (c.cohMinJ< cl.conceptList.get(i).cohMinJ){
+				rightNeighborsNumber++;
+				sum+=1.*c.cohMinJ/cl.conceptList.get(i).cohMinJ;
+			}
+		}
+		if (rightNeighborsNumber == 0)
+			return 0;
+		else{
+			//truthDegree = rightNeighborsNumber/tetta/lowerNeighbors.size();
+			truthDegree = (1.*rightNeighborsNumber/lowerNeighbors.size()>=tetta) ? 1 : 0;
+			truthDegree =  (float) Math.min(1.0, truthDegree);
+			return (sum/rightNeighborsNumber)*truthDegree;
+		}			
+	}
+	
+	public double lowerCohMinByAvgJ(FormalConcept c,float tetta){
+		double min = Integer.MAX_VALUE,
+				val = 0;
+		int rightNeighborsNumber = 0;
+		float truthDegree = 0;
+		Set<Integer> lowerNeighbors =c.childs;
+		for (Integer i: lowerNeighbors){
+			if (c.cohAvgJ< cl.conceptList.get(i).cohAvgJ){
+				rightNeighborsNumber++;
+				val = 1.*c.cohAvgJ/cl.conceptList.get(i).cohAvgJ;
+				if (val<min)
+					min = val;	
+			}
+		}
+		if (rightNeighborsNumber == 0)
+			return 0;
+		else{
+			truthDegree = (1.*rightNeighborsNumber/lowerNeighbors.size()>=tetta) ? 1 : 0;
+			truthDegree =  (float) Math.min(1.0, truthDegree);
+			return (min)*truthDegree;
+		}		
+	}
+	
+	public double lowerCohMinByMinJ(FormalConcept c,float tetta){
+		double min = Integer.MAX_VALUE,
+				val = 0;
+		int rightNeighborsNumber = 0;
+		float truthDegree = 0;
+		Set<Integer> lowerNeighbors =c.childs;
+		for (Integer i: lowerNeighbors){
+			if (c.cohMinJ< cl.conceptList.get(i).cohMinJ){
+				rightNeighborsNumber++;
+				val = 1.*c.cohMinJ/cl.conceptList.get(i).cohMinJ;
+				if (val<min)
+					min = val;	
+			}
+		}
+		if (rightNeighborsNumber == 0)
+			return 0;
+		else{
+			truthDegree = (1.*rightNeighborsNumber/lowerNeighbors.size()>=tetta) ? 1 : 0;
+			truthDegree =  (float) Math.min(1.0, truthDegree);
+			return (min)*truthDegree;
+		}		
+	}
+	
+	public double lowerCohAvgByAvgSMC(FormalConcept c,float tetta){
+		double sum = 0;
+		int rightNeighborsNumber = 0;
+		float truthDegree = 0;
+		Set<Integer> lowerNeighbors =c.childs;
+		for (Integer i: lowerNeighbors){
+			if (c.cohAvgSMC < cl.conceptList.get(i).cohAvgSMC){
+				rightNeighborsNumber++;
+				sum+=1.*c.cohAvgSMC/cl.conceptList.get(i).cohAvgSMC;	
+			}
+		}
+		if (rightNeighborsNumber == 0)
+			return 0;
+		else{
+			truthDegree = (1.*rightNeighborsNumber/lowerNeighbors.size()>=tetta) ? 1 : 0;
+			truthDegree =  (float) Math.min(1.0, truthDegree);
+			return (sum/rightNeighborsNumber)*truthDegree;
+		}			
+	}
+	
+	public double lowerCohAvgByMinSMC(FormalConcept c,float tetta){
+		double sum = 0;
+		int rightNeighborsNumber = 0;
+		float truthDegree = 0;
+		Set<Integer> lowerNeighbors =c.childs;
+		for (Integer i: lowerNeighbors){
+			if (c.cohMinSMC < cl.conceptList.get(i).cohMinSMC){
+				rightNeighborsNumber++;
+				sum+=1.*c.cohMinSMC/cl.conceptList.get(i).cohMinSMC;		
+			}
+		}
+		if (rightNeighborsNumber == 0)
+			return 0;
+		else{
+			truthDegree = (1.*rightNeighborsNumber/lowerNeighbors.size()>=tetta) ? 1 : 0;
+			truthDegree =  (float) Math.min(1.0, truthDegree);
+			return (sum/rightNeighborsNumber)*truthDegree;
+		}			
+	}
+	
+	public double lowerCohMinByAvgSMC(FormalConcept c,float tetta){
+		double min = Integer.MAX_VALUE,
+				val = 0;
+		int rightNeighborsNumber = 0;
+		float truthDegree = 0;
+		Set<Integer> lowerNeighbors =c.childs;
+		for (Integer i: lowerNeighbors){
+			if (c.cohAvgSMC<=cl.conceptList.get(i).cohAvgSMC){
+				rightNeighborsNumber++;
+				val = 1.*c.cohAvgSMC/cl.conceptList.get(i).cohAvgSMC;	
+				if (val<min)
+					min = val;	
+			}
+		}
+		if (rightNeighborsNumber == 0)
+			return 0;
+		else{
+			truthDegree = (1.*rightNeighborsNumber/lowerNeighbors.size()>=tetta) ? 1 : 0;
+			truthDegree =  (float) Math.min(1.0, truthDegree);
+			return (min)*truthDegree;
+		}		
+	}
+	
+	
+	public double lowerCohMinByMinSMC(FormalConcept c,float tetta){
+		double min = Integer.MAX_VALUE,
+				val = 0;
+		int rightNeighborsNumber = 0;
+		float truthDegree = 0;
+		Set<Integer> lowerNeighbors =c.childs;
+		for (Integer i: lowerNeighbors){
+			if (c.cohMinSMC< cl.conceptList.get(i).cohMinSMC){
+				rightNeighborsNumber++;
+				val = 1.*c.cohMinSMC/cl.conceptList.get(i).cohMinSMC;
+				if (val<min)
+					min = val;	
+			}
+		}
+		if (rightNeighborsNumber == 0)
+			return 0;
+		else{
+			truthDegree = (1.*rightNeighborsNumber/lowerNeighbors.size()>=tetta) ? 1 : 0;
+			return (min)*truthDegree;
+		}		
+	}
+	
+	
+	//Similarity approach (S)
+	//group of metrics: 
+	// SMC avg_avg, avg_min, min_avg
+	//   J avg_avg, avg_min, min_avg
+	public void similarityGoguenNorm(){
+		
+		// \ufffd\ufffd\ufffd \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd \ufffd\ufffd \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd \ufffd\ufffd \ufffd\ufffd\ufffd\ufffd \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd \ufffd \ufffd\ufffd\ufffd\ufffd\ufffd
+		// \ufffd\ufffd\ufffd\ufffd\ufffd \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd  \ufffd \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd \ufffd \ufffd\ufffd\ufffd\ufffd\ufffd
+		double[] buf = new double[2];
+		
+		if (objectsSimilaritySMC == null)
+			this.setUp();		
+		
+	//upper neighbors - alpha2
+	//lower neighbors - alpha3
+		for (FormalConcept c: cl.conceptList){	
+			c.cohAvgJ = avgCohJ(c);
+			c.cohMinJ = minCohJ(c);
+			c.cohAvgSMC = avgCohSMC(c);
+			c.cohMinSMC = minCohSMC(c);
+			
+		}
+		for (FormalConcept c: cl.conceptList){
+			float tetta = 1;
+			c.blS_Jaa = (c.cohAvgJ != 0) ? c.cohAvgJ * upperCohAvgByAvgJ(c,tetta) * lowerCohAvgByAvgJ(c,tetta) : 0;
+			c.blS_Jma = (c.cohMinJ != 0) ? c.cohMinJ * upperCohAvgByMinJ(c,tetta) * lowerCohAvgByMinJ(c,tetta) : 0;			
+			c.blS_Jam = (c.cohAvgJ != 0) ? c.cohAvgJ * upperCohMinByAvgJ(c,tetta) * lowerCohMinByAvgJ(c,tetta) : 0;
+			c.blS_Jmm = (c.cohMinJ != 0) ? c.cohMinJ * upperCohMinByMinJ(c,tetta) * lowerCohMinByMinJ(c,tetta) : 0;
+			c.blS_SMCaa = (c.cohAvgSMC != 0) ? c.cohAvgSMC * upperCohAvgByAvgSMC(c,tetta) * lowerCohAvgByAvgSMC(c,tetta) : 0;
+			c.blS_SMCma = (c.cohMinSMC != 0) ? c.cohMinSMC * upperCohAvgByMinSMC(c,tetta) * lowerCohAvgByMinSMC(c,tetta) : 0;
+			c.blS_SMCam = (c.cohAvgSMC != 0) ? c.cohAvgSMC * upperCohMinByAvgSMC(c,tetta) * lowerCohMinByAvgSMC(c,tetta) : 0;
+			c.blS_SMCmm = (c.cohMinSMC != 0) ? c.cohMinSMC * upperCohMinByMinSMC(c,tetta) * lowerCohMinByMinSMC(c,tetta) : 0;
+		}	
+	}
+	
+	//Cue validity approach (CV)
+	public void cueValidity(){
+		
+		if (attributesExtent == null)
+				this.setUp();
+		
+		ArrayList<Integer> attrExtent;
+		Set<Integer> intersection;
+		double sum = 0;
+		for (FormalConcept c: cl.conceptList){
+			sum = 0;
+			for (Integer i: c.intent){
+				intersection = new HashSet<Integer>();
+				intersection.addAll(c.extent);
+				attrExtent = attributesExtent.get(i);				
+				intersection.retainAll(attrExtent);
+				sum+=(double)intersection.size()*1./attrExtent.size();
+				}	
+			c.blCV = Double.isNaN(sum) ? 0 : sum;
+			
+		}			
+	}
+	//Category feature collocation approach
+	public void categoryFeatureCollocation(){
+		if (attributesExtent == null)
+			this.setUp();
+		
+		ArrayList<Integer> attrExtent;
+		Set<Integer> intersection;
+		double sum = 0;
+		int latticeSize = cl.conceptList.size();
+		for (FormalConcept c: cl.conceptList){
+			sum = 0;
+			for (int i = 0; i < cl.attributeCount; i++){
+				intersection = new HashSet<Integer>();
+				intersection.addAll(c.extent);
+				attrExtent = attributesExtent.get(i);				
+				intersection.retainAll(attrExtent);
+				sum+=(double)intersection.size()*1./attrExtent.size()*intersection.size()/c.extent.size();
+				}	
+			c.blCFC = Double.isNaN(sum) ? 0 : sum;
+		}
+	}
+	
+	//Category utility approach
+	public void categoryUtility(){
+		if (attributesExtent == null)
+			this.setUp();
+		
+		ArrayList<Integer> attrExtent;
+		Set<Integer> intersection;
+		double sum = 0;
+		int attrSize = cl.objectCount;
+		int cExtentSize = 0;
+		for (FormalConcept c: cl.conceptList){
+			sum = 0;
+			for (int i = 0; i < cl.attributeCount; i++){
+				intersection = new HashSet<Integer>();
+				intersection.addAll(c.extent);
+				cExtentSize = c.extent.size();
+				attrExtent = attributesExtent.get(i);				
+				intersection.retainAll(attrExtent);
+				sum+=(double)Math.pow(intersection.size()*1./cExtentSize,2)-Math.pow(1.*attrExtent.size()/attrSize,2);
+				}	
+			c.blCU =Double.isNaN(1.*cExtentSize/attrSize*sum) ? 0 : 1.*cExtentSize/attrSize*sum;
+		}					
+	}
+	
+	//Predictability approach (P)
+	public void predictability(){
+		
+		if (attributesExtent == null)
+			this.setUp();
+		ArrayList<Integer> attributes = new ArrayList<Integer>();
+		ArrayList<Integer> outOfIntent = new ArrayList<Integer>();
+		Set<Integer> intersection;
+		ArrayList<Integer> attrExtent;
+		double sum, term;
+		
+		for (int i = 0; i< cl.attributeCount; i++){
+			attributes.add(i);
+		}
+		for (FormalConcept c: cl.conceptList){
+			sum = 0;
+			outOfIntent = new ArrayList<Integer>();
+			outOfIntent.addAll(attributes);
+			outOfIntent.removeAll(c.intent);
+			for (Integer y: outOfIntent){
+				intersection = new HashSet<Integer>();
+				intersection.addAll(c.extent);	
+				attrExtent = attributesExtent.get(y);				
+				intersection.retainAll(attrExtent);
+				term = 1.*intersection.size()/c.extent.size();
+				
+				if (term > 0){
+					sum-=term*Math.log(term);
+				}
+			}
+			c.blP = Double.isNaN(1-sum/outOfIntent.size()) ? 0 : 1-sum/outOfIntent.size();
+		}
+		
+	}
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/fca/ConceptLattice.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/fca/ConceptLattice.java b/opennlp-similarity/src/main/java/opennlp/tools/fca/ConceptLattice.java
new file mode 100755
index 0000000..6bd546c
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/fca/ConceptLattice.java
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.fca;
+
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.ListIterator;
+import java.util.Set;
+
+import org.apache.commons.collections.ListUtils;
+
+
+public class ConceptLattice {
+	int objectCount;
+	int attributeCount;
+	ArrayList<FormalConcept> conceptList;
+	int [][] binaryContext;
+	static Measures ms;
+	static BasicLevelMetrics blm;
+	
+	public ConceptLattice(int objCount, int attrCount, int [][] binaryContext,boolean stats) {
+		this.objectCount = objCount;
+		this.attributeCount = attrCount;
+		this.binaryContext = binaryContext;
+		this.conceptList = new ArrayList<FormalConcept>();
+		FormalConcept bottom = new FormalConcept();
+		bottom.setPosition(0);
+		conceptList.add(bottom);
+		ms = new Measures(this);
+		blm = new BasicLevelMetrics(this);		
+		if (this.binaryContext !=null)
+			createLatticeFromBinaryContext();
+		if (stats){
+			ms.logStabilityExt();
+			ms.separation();	
+			ms.probability();
+			blm.cueValidity();
+			blm.categoryFeatureCollocation();
+			blm.categoryUtility();
+			blm.predictability();
+			blm.similarityGoguenNorm();		
+		}
+	}
+	
+	public ConceptLattice(String filename, boolean stats) throws FileNotFoundException, IOException {
+		
+		FcaReader fr = new FcaReader();
+		fr.ReadContextFromCxt(filename);
+		this.objectCount = fr.getObjectsCount();
+		this.attributeCount = fr.getAttributesCount();
+		this.binaryContext = fr.getBinaryContext();
+		
+		this.conceptList = new ArrayList<FormalConcept>();
+		FormalConcept bottom = new FormalConcept();
+		bottom.setPosition(0);
+		conceptList.add(bottom);
+		ms = new Measures(this);
+		blm = new BasicLevelMetrics(this);
+		if (this.binaryContext !=null)
+			createLatticeFromBinaryContext();
+		if (stats){
+			ms.logStabilityExt();
+			ms.separation();	
+			ms.probability();
+			blm.cueValidity();
+			blm.categoryFeatureCollocation();
+			blm.categoryUtility();
+			blm.predictability();
+			blm.similarityGoguenNorm();
+		}				
+	}
+
+	
+	public int GetMaximalConcept(List<Integer> intent, int Generator) {
+		boolean parentIsMaximal = true;
+		while(parentIsMaximal) {
+			parentIsMaximal = false;
+			for (int parent : conceptList.get(Generator).getParents()) {
+				if (conceptList.get(parent).getIntent().containsAll(intent)) {
+					Generator = parent;
+					parentIsMaximal = true;
+					break;
+				}
+			}
+		}
+		return Generator;
+	}
+	
+	public void AddExtentToAncestors(LinkedHashSet<Integer>extent, int curNode) {
+		if (conceptList.get(curNode).parents.size()>0){
+			for (int parent : conceptList.get(curNode).parents){
+				conceptList.get(parent).addExtents(extent);
+				AddExtentToAncestors(extent, parent);
+			}
+		}	
+	}
+	
+	public int AddIntent(List<Integer> intent,LinkedHashSet<Integer>extent, int generator) {
+		//System.out.println("add intent "+intent+extent+generator);
+		int generator_tmp = GetMaximalConcept(intent, generator);	
+		generator = generator_tmp;
+		//System.out.println("	max gen "+generator);
+		if (conceptList.get(generator).getIntent().equals(intent)) {
+			conceptList.get(generator).addExtents(extent);
+			AddExtentToAncestors(extent, generator);							
+			return generator;
+		}
+		Set<Integer> generatorParents = conceptList.get(generator).getParents();
+		Set<Integer> newParents = new HashSet<Integer>();
+		for (int candidate : generatorParents) {
+			if (!intent.containsAll(conceptList.get(candidate).getIntent())) {
+				List<Integer> intersection = ListUtils.intersection(intent, conceptList.get(candidate).getIntent());				
+				LinkedHashSet<Integer> new_extent = new LinkedHashSet<Integer>();
+				new_extent.addAll(conceptList.get(candidate).extent);
+				new_extent.addAll(extent);
+				candidate = AddIntent(intersection,new_extent,candidate);
+			}
+			
+			boolean addParents = true;
+			Iterator<Integer> iterator = newParents.iterator();
+			while (iterator.hasNext()) {
+				Integer parent = iterator.next();
+				if (conceptList.get(parent).getIntent().containsAll(conceptList.get(candidate).getIntent())) {
+					addParents = false;
+					break;
+				}
+				else {
+					if (conceptList.get(candidate).getIntent().containsAll(conceptList.get(parent).getIntent())) {
+						iterator.remove();
+					}
+				}
+			}
+			if (addParents) {
+				newParents.add(candidate);
+			}
+		}
+		
+		FormalConcept newConcept = new FormalConcept();
+		newConcept.setIntent(intent);
+		LinkedHashSet<Integer> new_extent = new LinkedHashSet<Integer>();
+		new_extent.addAll(conceptList.get(generator).extent);
+		new_extent.addAll(extent);
+		newConcept.addExtents(new_extent);
+		newConcept.setPosition(conceptList.size());
+		conceptList.add(newConcept);
+		conceptList.get(generator).getParents().add(newConcept.position);
+		conceptList.get(newConcept.position).childs.add(generator);
+		for (int newParent: newParents) {
+			if (conceptList.get(generator).getParents().contains(newParent)) {
+				conceptList.get(generator).getParents().remove(newParent);
+				conceptList.get(newParent).childs.remove(generator);
+			}
+			conceptList.get(newConcept.position).getParents().add(newParent);
+			conceptList.get(newParent).addExtents(new_extent);
+			AddExtentToAncestors(new_extent, newParent);
+			conceptList.get(newParent).childs.add(newConcept.position);
+		}
+		
+		return newConcept.position;
+	}
+	
+	public void printLatticeStats() {
+		System.out.println("Lattice stats");
+		System.out.println("max_object_index = " + objectCount);
+		System.out.println("max_attribute_index = " + attributeCount);
+		System.out.println("Current concept count = " + conceptList.size());
+	}
+	
+	public void printLattice() {
+		for (int i = 0; i < conceptList.size(); ++i) {
+			printConceptByPosition(i);
+		}
+	}
+	
+	public void printLatticeFull() {
+		for (int i = 0; i < conceptList.size(); ++i) {
+			printConceptByPositionFull(i);
+		}
+	}
+	
+	public void printContext() {
+		for (int i = 0; i<objectCount; i++){
+			for (int j = 0; j<attributeCount; j++){
+				System.out.print(binaryContext[i][j]+" ");
+			}
+			System.out.println();
+		}	
+	}
+	
+	public void printConceptByPosition(int index) {
+		System.out.println("Concept at position " + index);
+		conceptList.get(index).printConcept();
+	}
+	
+	public void printConceptByPositionFull(int index) {
+		System.out.println("Concept at position " + index);
+		conceptList.get(index).printConceptFull();
+	}
+	
+	public void createLatticeFromBinaryContext(){
+		LinkedHashSet<Integer> obj;
+		ArrayList<Integer> intent;
+		// attributes list
+		ArrayList<Integer> attributes = new ArrayList<Integer>();
+		for (int i = 0; i <attributeCount; i++){
+			attributes.add(i);
+		}
+		// objects set
+		LinkedHashSet<Integer> objects = new LinkedHashSet<Integer>();
+		for (int i = 0; i <objectCount; i++){
+			objects.add(i);
+		}
+		
+		this.conceptList.get(0).setIntent(attributes);
+		for (int i = 0; i < objectCount; i++){
+			intent = new ArrayList<Integer>();
+			obj = new LinkedHashSet<Integer>();
+			obj.add(i);
+			for (int j = 0; j < attributeCount; j++){
+				if (binaryContext[i][j] == 1){
+					intent.add(j);
+				}
+			}
+			this.AddIntent(intent,obj,0);
+		}	
+	}
+	
+	public static void main(String []args) throws FileNotFoundException, IOException {
+
+		ConceptLattice cl = new ConceptLattice("sports.cxt", true);
+		cl.printLattice();	
+	}
+	
+	
+	public List<Integer> getAttributeExtByID(int ind){
+		ArrayList<Integer> attrExt = new ArrayList<Integer>();
+		for (int i=0;i<objectCount; i++)
+			if (binaryContext[i][ind]==1)
+				attrExt.add(i); 
+		return attrExt;
+	}
+	
+	public ArrayList<Integer> getObjectIntByID(int ind){
+		ArrayList<Integer> objInt = new ArrayList<Integer>();
+		for (int i=0;i<attributeCount; i++)
+			if (binaryContext[ind][i]==1)
+				objInt.add(i); 
+		return objInt;
+	}	
+	
+	public ArrayList<FormalConcept> getLattice(){
+		return conceptList;
+	}
+	
+	public int getAttributesCount() {
+		return attributeCount;
+	}
+	
+	public int getObjectCount() {
+		return objectCount;
+	}
+	
+	public int getSize(){
+		return conceptList.size();
+	}
+
+	
+	public void printBinContext() {
+		for (int i = 0; i < binaryContext.length; i++ ){
+				System.out.println(Arrays.toString(binaryContext[i]));
+		}	
+	}
+
+
+	
+}
+

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/fca/FcaConverter.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/fca/FcaConverter.java b/opennlp-similarity/src/main/java/opennlp/tools/fca/FcaConverter.java
new file mode 100755
index 0000000..b1eaa93
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/fca/FcaConverter.java
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.fca;
+
+import java.util.ArrayList;
+import java.util.Set;
+
+public class FcaConverter {
+	
+	public FcaConverter (){
+		
+	}
+	
+	public int [][] latticeToContext(ConceptLattice cl){
+		
+		int newAttrCount = cl.conceptList.size();
+		int [][] binaryContext = null;
+		ArrayList<FormalConcept> cList = new ArrayList<FormalConcept>();
+		cList.addAll(cl.conceptList);
+		boolean run = true;
+		int k=0;
+		while (run && k<cl.conceptList.size()){
+			if (cl.conceptList.get(k).intent.size() == cl.attributeCount){
+					for (Integer i:cl.conceptList.get(k).parents){
+						//cList.remove(cl.conceptList.get(i));
+					}
+				//cList.remove(cl.conceptList.get(k));
+				run=false;
+			}
+			else{
+				//cList.add(arg0, arg1);
+				
+			}
+			
+		}
+		//System.out.println("cList.size() " + cList.size());
+		run = true;
+		k=0;
+		while (run && k<=newAttrCount){
+			if (cList.get(k).extent.size()==0)
+				k++;
+				run = false;
+		}
+		newAttrCount = cList.size();
+		Set<Integer> nodeExtend;
+		binaryContext = new int[cl.objectCount][newAttrCount];
+		for (int j = 0; j<newAttrCount; j++){
+			nodeExtend = cList.get(j).extent;
+			//System.out.println(cList.get(j).position+" nodeExtend " + nodeExtend);
+			for (Integer i: nodeExtend){
+				binaryContext[i][j]=1;
+			}
+		}
+		return binaryContext;
+	}
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/fca/FcaReader.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/fca/FcaReader.java b/opennlp-similarity/src/main/java/opennlp/tools/fca/FcaReader.java
new file mode 100755
index 0000000..cca4e6a
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/fca/FcaReader.java
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.fca;
+
+import java.io.BufferedReader;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+
+public class FcaReader {
+	
+	ArrayList<String> obNames = null;
+	ArrayList<String> atNames = null;
+	int [][] binContext = null;
+	int objectsNumber = 0;
+	int attributesNumber = 0;
+	
+	public void ReadContextFromCxt(String filename) throws FileNotFoundException, IOException{
+
+		obNames = new ArrayList<String>();
+		atNames = new ArrayList<String>();
+					
+		BufferedReader br = new BufferedReader(new FileReader(filename));
+		try	{
+		    String line;		    
+		    br.readLine(); //B
+		    br.readLine();
+		    objectsNumber = Integer.parseInt(br.readLine());
+		    attributesNumber = Integer.parseInt(br.readLine());
+		    br.readLine();	
+	    
+		    binContext = new int [objectsNumber][attributesNumber];
+		    
+		    for (int i=0;i<objectsNumber;i++){
+		    	obNames.add(br.readLine());
+		    }
+		    
+		    for (int i=0;i<attributesNumber;i++){
+		    	atNames.add(br.readLine());
+		    }
+		    
+		    int i=0;
+		    while ((line = br.readLine()) != null) {
+		    	for (int j = 0; j<line.length();j++){
+		    		if (line.charAt(j)=='.'){
+		    			binContext[i][j] = 0;
+		    		}		    			
+		    		else  
+			    		binContext[i][j] = 1;
+			    }
+		    	i+=1;	    	
+		    }	    
+		} catch (Exception e){
+			e.printStackTrace();
+		}
+	}
+	
+	public int getAttributesCount(){
+		return attributesNumber;
+	}
+	
+	public int getObjectsCount(){
+		return objectsNumber;
+	}
+	
+	public int[][] getBinaryContext(){
+		return binContext;
+	}
+	
+	
+	public static void main(String []args) throws FileNotFoundException, IOException{
+		
+		FcaReader loader = new FcaReader();
+		loader.ReadContextFromCxt("C://Users/Tanya/Desktop/\ufffd\ufffd\ufffd\ufffd\ufffd/1 \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd/\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd \ufffd \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd \ufffd\ufffd\ufffd\ufffd\ufffd\ufffd/\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd\ufffd/sports.cxt");
+		
+	}
+	
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/fca/FcaWriter.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/fca/FcaWriter.java b/opennlp-similarity/src/main/java/opennlp/tools/fca/FcaWriter.java
new file mode 100755
index 0000000..10a4851
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/fca/FcaWriter.java
@@ -0,0 +1,131 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.fca;
+
+import java.io.BufferedWriter;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.Writer;
+
+public class FcaWriter {
+	
+	public void WriteAsCxt(String filename, ConceptLattice cl){
+		
+		Writer writer = null;
+
+		try {
+		    writer = new BufferedWriter(new OutputStreamWriter(
+		          new FileOutputStream(filename), "utf-8"));
+		    writer.write("B\n");
+		    writer.write("\n");
+		    writer.write(String.valueOf(cl.objectCount)+"\n");
+		    writer.write(cl.attributeCount + "\n");
+		    writer.write("\n");		    
+		    
+		    for (int obj = 0; obj < cl.objectCount; obj++){
+		    	writer.write(obj + "\n");
+		    }	    
+		    for (int attr = 0; attr < cl.attributeCount; attr++){
+		    	writer.write(attr + "\n");
+		    }
+		    
+		    for (int i = 0; i < cl.objectCount; i++){
+		    	for (int j = 0; j < cl.attributeCount; j++){
+		    		writer.write((cl.binaryContext[i][j] == 0) ? '.':'X');
+		    	}
+		    	writer.write("\n");
+		    }
+		    
+		    writer.close();
+		    
+		} catch (IOException ex) {
+			System.err.println(ex.getMessage());
+		} finally {
+		   try {writer.close();} catch (Exception ex) {}
+		}
+	}
+	
+	public void WriteStatsToTxt(String filename, ConceptLattice cl, int patternStructureSize){
+		///2n+1
+		int intentSize = 2*cl.attributeCount+1; //4*cl.ob+1;
+		String formatStrHeader = "%-9s %-9s %-13s %-9s %-9s %-9s %-9s %-9s %-9s %-9s %-9s %-9s %-9s %-9s %-9s %-"+ intentSize +"s%n";
+		String formatStr = "%6.3f %6.3f [%5.3f; %5.3f] %6.3f %6.3f %6.3f %6.3f %6.3f %6.3f %6.3f %6.3f %6.3f %6.3f %6.3f %6.3f %"+ intentSize +"s%n";
+		String formatStr2 = "%9.2f %9.2f [%5.2f; %5.2f] %9.2f %9.2f %9.2f %9.2f %9.2f %9.2f %9.2f %9.2f %9.2f %9.2f %9.2f %9.2f %-"+ intentSize +"s%n";
+		Writer writer = null;
+
+		try {
+		    writer = new BufferedWriter(new OutputStreamWriter(
+		          new FileOutputStream(filename), "utf-8"));
+		    writer.write("Lattice size: " + cl.getSize() +", obj: " + cl.objectCount  + ", attr: " + cl.attributeCount +"\n");
+		    writer.write("PatternStructure size: " + patternStructureSize + "\n");
+		    writer.write(String.format(formatStrHeader, "probability", "separation", "stability", "blCV", "blCFC", "blCU", "blP", "blS_SMCaa", "blS_SMCam", "blS_SMCma", "blS_SMCma", "blS_Jaa",  "blS_Jam",  "blS_Jma", "blS_Jmm", "extent"));			 
+	    	
+		    for (FormalConcept c : cl.conceptList){
+		    	writer.write(String.format(formatStr2, c.probability, c.separation, c.intLogStabilityBottom, c.intLogStabilityUp, c.blCV, c.blCFC, c.blCU, c.blP, 
+		    			c.blS_SMCaa,  c.blS_SMCam, c.blS_SMCma, c.blS_SMCmm, c.blS_Jaa,  c.blS_Jam,  c.blS_Jma, c.blS_Jmm, c.extent));
+		    }
+
+		    writer.close();
+		    
+		} catch (IOException ex) {
+			System.err.println(ex.getMessage());
+		} finally {
+		   try {writer.close();} catch (Exception ex) {}
+		}
+	}
+	
+	
+	public void WriteStatsToCvs(String filename, ConceptLattice cl, int patternStructureSize){
+		///2n+1
+		int intentSize = 2*cl.attributeCount+1; //4*cl.ob+1;
+		String formatStrHeader = "%s; %s; %s; %s; %s; %s; %s; %s; %s; %s; %s; %s; %s; %s; %s; %s%n";
+		String formatStr2 = "%.2f; %.2f; [%5.2f : %5.2f]; %.2f; %.2f; %.2f; %.2f; %.2f; %.2f; %.2f; %.2f; %.2f; %.2f; %.2f; %.2f; %s%n";
+		Writer writer = null;
+
+		try {
+		    writer = new BufferedWriter(new OutputStreamWriter(
+		          new FileOutputStream(filename), "utf-8"));
+		    writer.write("Lattice size: " + cl.getSize() +", obj: " + cl.objectCount  + ", attr: " + cl.attributeCount +"\n");
+		    writer.write("PatternStructure size: " + patternStructureSize + "\n");
+		    writer.write(String.format(formatStrHeader, "probability", "separation", "stability", "blCV", "blCFC", "blCU", "blP", "blS_SMCaa", "blS_SMCam", "blS_SMCma", "blS_SMCma", "blS_Jaa",  "blS_Jam",  "blS_Jma", "blS_Jmm", "extent"));			 
+	    	
+		    for (FormalConcept c : cl.conceptList){
+		    	writer.write(String.format(formatStr2, c.probability, c.separation, c.intLogStabilityBottom, c.intLogStabilityUp, c.blCV, c.blCFC, c.blCU, c.blP, 
+		    			c.blS_SMCaa,  c.blS_SMCam, c.blS_SMCma, c.blS_SMCmm, c.blS_Jaa,  c.blS_Jam,  c.blS_Jma, c.blS_Jmm, c.extent));
+		    }
+
+		    writer.close();
+		    
+		} catch (IOException ex) {
+			 System.err.println(ex.getMessage());
+		} finally {
+		   try {writer.close();} catch (Exception ex) {}
+		}
+	}
+	
+	
+public static void main(String []args) throws FileNotFoundException, IOException{
+	
+	ConceptLattice cl = new ConceptLattice("sports.cxt",false);	
+	FcaWriter writer = new FcaWriter();
+	writer.WriteAsCxt("res.cxt",cl);
+		
+	}
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/fca/FormalConcept.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/fca/FormalConcept.java b/opennlp-similarity/src/main/java/opennlp/tools/fca/FormalConcept.java
new file mode 100755
index 0000000..3f5ecb8
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/fca/FormalConcept.java
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.fca;
+
+import java.util.ArrayList;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+
+public class FormalConcept {
+	int position;
+	ArrayList<Integer>intent;
+	Set<Integer> childs;
+	Set<Integer> parents;
+	Set<Integer> extent;
+	double intLogStabilityBottom = 0,
+			intLogStabilityUp = 0,
+			separation = 0,
+			probability = 0,
+			blS_Jaa = 0,	
+			blS_SMCaa = 0,
+			blS_Jam = 0,	
+			blS_SMCam = 0,
+			blS_Jmm = 0,	
+			blS_Jma = 0,	
+			blS_SMCma = 0,
+			blS_SMCmm = 0,
+			blCV = 0,
+			blCFC = 0,
+			blCU = 0,
+			blP = 0;
+	
+	double cohAvgJ = 0,
+		   cohMinJ = 0,
+		   cohAvgSMC = 0,
+		   cohMinSMC = 0;
+	
+	
+	public FormalConcept() {
+		position = -1;
+		intent = new ArrayList<Integer>();
+		extent = new LinkedHashSet<Integer>();
+		parents = new HashSet<Integer>();
+		childs = new HashSet<Integer>();	
+	}
+
+	public void setPosition(int newPosition ){
+	       position = newPosition;
+	}
+
+	public void addExtents(LinkedHashSet<Integer> ext){
+		extent.addAll(ext);
+	}
+	
+	public LinkedHashSet<Integer> getExtent(){
+		return (LinkedHashSet<Integer>) extent;
+	}
+	
+	public void setIntent(List<Integer> newIntent ){
+	       intent.clear();
+	       intent.addAll(newIntent);
+	}
+	
+	public void setChilds(Set<Integer> newChilds ){
+		childs.clear();
+		childs.addAll(newChilds);
+	}
+	
+	public void addChild(Integer child ){
+		childs.add(child);
+	}
+	
+	public void setParents( Set<Integer> newParents ){
+	       //parents = newParents;
+		parents.clear();
+		parents.addAll(newParents);
+	}
+	
+	public void printConcept() {
+		System.out.println("Concept position:" + position);
+		System.out.println("Concept intent:" + getIntent());
+		System.out.println("Concept extent:" + extent);
+		System.out.println("Concept parents:" + getParents());
+		System.out.println("Concept childs:" + childs);
+	    System.out.println("--------------------");
+	}
+	
+	public void printConceptFull() {
+		System.out.println("Concept position:" + position);
+		System.out.println("Concept intent:" + getIntent());
+		System.out.println("Concept extent:" + extent);
+		System.out.println("Concept parents:" + getParents());
+		System.out.println("Concept childs:" + childs);
+		System.out.format("Prob.  blSaaJ   blSaaSMC blSmaJ   blSmmJ   blSmaSMC  blSamJ   blSamSMC  blCV     blCFC    blCU     blP      separ.\n");
+		System.out.format("%5.3f %8.3f %8.3f %8.3f %8.3f %8.3f %8.3f %8.3f %8.3f %8.3f %8.3f %8.3f %8.3f\n", probability, blS_Jaa, blS_SMCaa,
+				 blS_Jma, blS_Jmm, blS_SMCma, blS_Jam, blS_SMCam, blCV, blCFC, blCU, blP, separation);	
+		System.out.format("Concept cohAvgJ: %.3f \n", cohAvgJ);
+		System.out.format("Concept cohAvgSMC: %.3f \n", cohAvgSMC);
+		System.out.format("Concept cohMinJ: %.3f \n", cohMinJ);
+		System.out.format("Concept cohMinSMC: %.3f \n", cohMinSMC);
+	    //System.out.format("Concept stability: [ %.3f; %.3f] \n", getIntLogStabilityBottom(), getIntLogStabilityUp());
+	    System.out.println("--------------------");
+	}
+	
+	 public static void main(String []args) {
+		 FormalConcept c = new FormalConcept();
+		 c.printConcept();
+	 }
+
+	public ArrayList<Integer> getIntent() {
+		return intent;
+	}
+
+	public void setIntent(ArrayList<Integer> intent) {
+		this.intent = intent;
+	}
+
+	public Set<Integer> getParents() {
+		return parents;
+	}
+
+	public double getIntLogStabilityBottom() {
+		return intLogStabilityBottom;
+	}
+
+	public void setIntLogStabilityBottom(double intLogStabilityBottom) {
+		this.intLogStabilityBottom = intLogStabilityBottom;
+	}
+
+	public double getIntLogStabilityUp() {
+		return intLogStabilityUp;
+	}
+
+	public void setIntLogStabilityUp(double intLogStabilityUp) {
+		this.intLogStabilityUp = intLogStabilityUp;
+	}
+}
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/fca/Measures.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/fca/Measures.java b/opennlp-similarity/src/main/java/opennlp/tools/fca/Measures.java
new file mode 100755
index 0000000..a4b17d8
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/fca/Measures.java
@@ -0,0 +1,147 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.fca;
+
+
+import org.apache.commons.math3.special.*;
+import java.util.ArrayList;
+import java.util.Set;
+
+public class Measures {
+		
+	ConceptLattice cl;
+	
+	public Measures(ConceptLattice cl) {
+		this.cl = cl;
+	}
+	
+	public void logStabilityInt(){
+		int min_delta = -1, delta = -1;
+		float sum = 0;
+		for (int i = 0; i < cl.conceptList.size(); ++i) {
+			min_delta = cl.attributeCount;
+			sum = 0;
+			FormalConcept fc = cl.conceptList.get(i);
+			Set<Integer> parents = fc.parents;
+			for (Integer j: parents) {
+				delta = fc.getIntent().size() - cl.conceptList.get(j).intent.size();
+				if (delta<min_delta)
+					min_delta = delta;
+				sum += Math.pow(2, -delta);
+			}
+			fc.intLogStabilityBottom=-(Math.log(sum)/Math.log(2.0));
+			fc.intLogStabilityUp = min_delta;
+		}
+	}
+	
+	public void logStabilityExt(){
+		int min_delta = -1, delta = -1;
+		float sum = 0;
+		for (int i = 0; i < cl.conceptList.size(); ++i) {
+			min_delta = cl.attributeCount;
+			sum = 0;
+			FormalConcept fc = cl.conceptList.get(i);
+			Set<Integer> childs = fc.childs;
+			for (Integer j: childs) {
+				delta = fc.getExtent().size() - cl.conceptList.get(j).getExtent().size();
+				if (delta<min_delta)
+					min_delta = delta;
+				sum += Math.pow(2, -delta);
+			}
+			fc.intLogStabilityBottom=-(Math.log(sum)/Math.log(2.0));
+			fc.intLogStabilityUp = min_delta;
+		}
+	}
+	
+	public void separation(){
+		ArrayList<Integer> intent;
+		Set<Integer> extentMembers;
+		int extentVolume = 0, intentVolume = 0;
+		double sz = 0;
+		for (int i = 0; i < cl.conceptList.size(); ++i) {
+			intent = cl.conceptList.get(i).getIntent();
+			extentMembers = cl.conceptList.get(i).extent;
+			extentVolume = 0;
+			intentVolume = 0;
+			for (Integer ext : extentMembers){
+				for (int attr = 0; attr<cl.attributeCount;attr++){
+					extentVolume += cl.binaryContext[ext][attr];
+				}				
+			}
+			for (int attr = 0; attr<intent.size();attr++ ){
+				for (int obj = 0; obj < cl.objectCount; obj++){
+					intentVolume += cl.binaryContext[obj][intent.get(attr)];
+				}				
+			}			
+			sz = intent.size()*extentMembers.size();
+			if (extentVolume+extentVolume-sz!=0)
+				cl.conceptList.get(i).separation = sz/(extentVolume+intentVolume-sz);
+			else
+				cl.conceptList.get(i).separation = 0;
+		}
+	}
+	public double attributeProbability(int attrNmb){
+	    double pAttr = 0;
+	    for (int i = 0; i<cl.objectCount;i++){
+	    	pAttr+=cl.binaryContext[i][attrNmb];
+	    }
+	    pAttr/=cl.objectCount;
+	    return pAttr;
+	}
+	
+	public double intentProbability(ArrayList<Integer> intent){
+		double pB = 1;
+	    for (int i=0;i<intent.size();i++){
+	    	pB*=attributeProbability(intent.get(i));
+	    }
+	    return pB;
+	}
+	
+	public void probability(){			
+		//the probability of B being closed
+		for (int i = 0; i<cl.conceptList.size();i++){
+			ArrayList<Integer> intent = cl.conceptList.get(i).getIntent();
+			// out of concept intent
+			double pB = intentProbability(intent);
+			ArrayList<Integer> outOfIntent = new ArrayList<Integer>();
+			ArrayList<Double> outOfIntentAttrProb = new ArrayList<Double>();
+			for (int j=0;j<cl.attributeCount; j++){
+				outOfIntent.add(j);
+			}
+			for (int j=intent.size()-1;j>=0;j--){
+				outOfIntent.remove(intent.get(j));
+			}
+			for (int j=0;j<outOfIntent.size();j++){
+				outOfIntentAttrProb.add(attributeProbability(outOfIntent.get(j)));
+			}
+			double prob = 0,  mult = 1, mult1=1;
+			int n = cl.objectCount;
+			for (int k=0; k<= n; k++){
+				mult = 1;
+				mult1 = 1;
+				for (int j=0;j<outOfIntentAttrProb.size();j++){
+					mult*=(1-Math.pow(outOfIntentAttrProb.get(j),k));
+				}				
+				mult1 = Math.pow(pB,k)*Math.pow(1-pB,n-k);
+				prob+=mult1*mult*Gamma.digamma(n+1)/Gamma.digamma(k+1)/Gamma.digamma(n-k+1);				
+			}
+			
+			cl.conceptList.get(i).probability = prob;			
+		}
+	}
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/fca/RandomNoiseGenerator.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/fca/RandomNoiseGenerator.java b/opennlp-similarity/src/main/java/opennlp/tools/fca/RandomNoiseGenerator.java
new file mode 100755
index 0000000..ff5a24f
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/fca/RandomNoiseGenerator.java
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package opennlp.tools.fca;
+
+import java.util.Random;
+
+public class RandomNoiseGenerator {
+	
+	public int [][] AlterCellsWithProbability(double p, int [][] binaryContext){
+		
+		Random rnd = new Random();
+		int n = binaryContext.length;
+		int m = binaryContext[0].length;
+		
+		for (int i = 0; i < n; i++ )
+			for (int j = 0; j < m; j++)
+				if (rnd.nextFloat() <= p){
+					if (binaryContext[i][j] == 0)
+						binaryContext[i][j] = 1;
+					else 
+						binaryContext[i][j] = 0;
+				}
+		return binaryContext;
+	}
+	
+	public int[][] AddObjectsAttributesWithProbability(int numberObjAttr, double d, int [][] binaryContext){
+		
+		Random rnd = new Random();
+		int nRandObjs = rnd.nextInt(numberObjAttr);
+		int nRandAttr = numberObjAttr - nRandObjs;
+		
+		int [][] newBinaryContext = new int [binaryContext.length + nRandObjs][binaryContext[0].length + nRandAttr];
+		for (int i = 0; i < binaryContext.length; i++)
+			for (int  j = 0; j < binaryContext[0].length; j++)
+				newBinaryContext[i][j] = binaryContext[i][j];
+		
+		for (int i = binaryContext.length; i < binaryContext.length + nRandObjs; i++)
+			for (int  j = 0; j < binaryContext[0].length + nRandAttr; j++)
+				newBinaryContext[i][j] = (rnd.nextFloat() <= d) ? 1 : 0;
+		
+		for (int  j = binaryContext[0].length; j < binaryContext[0].length + nRandAttr; j++)
+			for (int i = 0; i < binaryContext.length; i++)
+				newBinaryContext[i][j] = (rnd.nextFloat() <= d) ? 1 : 0;
+		
+		return newBinaryContext;
+	}
+}
+
+	
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/jsmlearning/JSMDecision.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/jsmlearning/JSMDecision.java b/opennlp-similarity/src/main/java/opennlp/tools/jsmlearning/JSMDecision.java
new file mode 100644
index 0000000..f451b39
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/jsmlearning/JSMDecision.java
@@ -0,0 +1,78 @@
+package opennlp.tools.jsmlearning;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+
+public class JSMDecision {
+	String keywordClassName;
+	Boolean bPositiveClass;
+	List<List<List<ParseTreeChunk>>> posHypotheses,
+			negHypotheses;
+	List<List<List<ParseTreeChunk>>> posIntersectionsUnderNeg , 
+			negIntersectionsUnderPos;
+	private String[] separationKeywords;
+	
+	public String[] getSeparationKeywords() {
+		return separationKeywords;
+	}
+	public void setSeparationKeywords(String[] separationKeywords) {
+		this.separationKeywords = separationKeywords;
+	}
+	public String getKeywordClassName() {
+		return keywordClassName;
+	}
+	public void setKeywordClassName(String keywordClassName) {
+		this.keywordClassName = keywordClassName;
+	}
+	public Boolean getbPositiveClass() {
+		return bPositiveClass;
+	}
+	public void setbPositiveClass(Boolean bPositiveClass) {
+		this.bPositiveClass = bPositiveClass;
+	}
+	public List<List<List<ParseTreeChunk>>> getPosHypotheses() {
+		return posHypotheses;
+	}
+	public void setPosHypotheses(List<List<List<ParseTreeChunk>>> posHypotheses) {
+		this.posHypotheses = posHypotheses;
+	}
+	public List<List<List<ParseTreeChunk>>> getNegHypotheses() {
+		return negHypotheses;
+	}
+	public void setNegHypotheses(List<List<List<ParseTreeChunk>>> negHypotheses) {
+		this.negHypotheses = negHypotheses;
+	}
+	public List<List<List<ParseTreeChunk>>> getPosIntersectionsUnderNeg() {
+		return posIntersectionsUnderNeg;
+	}
+	public void setPosIntersectionsUnderNeg(
+			List<List<List<ParseTreeChunk>>> posIntersectionsUnderNeg) {
+		this.posIntersectionsUnderNeg = posIntersectionsUnderNeg;
+	}
+	public List<List<List<ParseTreeChunk>>> getNegIntersectionsUnderPos() {
+		return negIntersectionsUnderPos;
+	}
+	public void setNegIntersectionsUnderPos(
+			List<List<List<ParseTreeChunk>>> negIntersectionsUnderPos) {
+		this.negIntersectionsUnderPos = negIntersectionsUnderPos;
+	}
+	public JSMDecision(String keywordClassName, Boolean bPositiveClass,
+			List<List<List<ParseTreeChunk>>> posHypotheses,
+			List<List<List<ParseTreeChunk>>> negHypotheses,
+			List<List<List<ParseTreeChunk>>> posIntersectionsUnderNeg,
+			List<List<List<ParseTreeChunk>>> negIntersectionsUnderPos, String[] separationKeywords) {
+		super();
+		this.keywordClassName = keywordClassName;
+		this.bPositiveClass = bPositiveClass;
+		this.posHypotheses = posHypotheses;
+		this.negHypotheses = negHypotheses;
+		this.posIntersectionsUnderNeg = posIntersectionsUnderNeg;
+		this.negIntersectionsUnderPos = negIntersectionsUnderPos;
+		this.separationKeywords = separationKeywords;
+	}
+	
+	
+
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/jsmlearning/JSMLearnerOnLatticeBase.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/jsmlearning/JSMLearnerOnLatticeBase.java b/opennlp-similarity/src/main/java/opennlp/tools/jsmlearning/JSMLearnerOnLatticeBase.java
new file mode 100644
index 0000000..52154cd
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/jsmlearning/JSMLearnerOnLatticeBase.java
@@ -0,0 +1,338 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.jsmlearning;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+import opennlp.tools.parse_thicket.pattern_structure.LinguisticPatternStructure;
+import opennlp.tools.similarity.apps.utils.Pair;
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+import opennlp.tools.textsimilarity.ParseTreeChunkListScorer;
+import opennlp.tools.textsimilarity.ParseTreeMatcherDeterministic;
+import opennlp.tools.textsimilarity.chunker2matcher.ParserChunker2MatcherProcessor;
+
+public class JSMLearnerOnLatticeBase {
+	ParserChunker2MatcherProcessor chunk_maker = ParserChunker2MatcherProcessor.getInstance();
+	LinguisticPatternStructure psPos = new LinguisticPatternStructure(0,0), psNeg = new LinguisticPatternStructure(0,0);
+	ParseTreeMatcherDeterministic md = new ParseTreeMatcherDeterministic(); 
+	
+
+
+
+	public JSMDecision buildLearningModel(List<String> posTexts, List<String> negTexts, 
+			String unknown, String[] separationKeywords){
+		psPos = new LinguisticPatternStructure(0,0); psNeg = new LinguisticPatternStructure(0,0);
+
+		if (separationKeywords!=null){ // re-sort by occurrence of separation keyword
+			Pair<List<String>, List<String>> pair = reGroupByOccurrenceOfSeparationKeyword(posTexts, negTexts, separationKeywords );
+			posTexts = pair.getFirst(); negTexts = 	pair.getSecond();
+		}
+
+		List<List<List<ParseTreeChunk>>> lingRepsPos = new ArrayList<List<List<ParseTreeChunk>>>(),
+				lingRepsNeg = new ArrayList<List<List<ParseTreeChunk>>>();
+		for(String text: posTexts)
+			lingRepsPos.add(chunk_maker.formGroupedPhrasesFromChunksForPara(text));
+
+		for(String text: negTexts)
+			lingRepsNeg.add(chunk_maker.formGroupedPhrasesFromChunksForPara(text));
+
+		LinkedHashSet<Integer> obj = null;
+		int i=0;
+		for(List<List<ParseTreeChunk>> chunk: lingRepsPos){
+			obj = new LinkedHashSet<Integer>();
+			obj.add(i);
+			psPos.AddIntent(chunk, obj, 0);
+			i++;
+		}
+		i=0;
+		for(List<List<ParseTreeChunk>> chunk: lingRepsNeg){
+			obj = new LinkedHashSet<Integer>();
+			obj.add(i);
+			psNeg.AddIntent(chunk, obj, 0);
+			i++;
+		}
+
+
+
+		List<List<ParseTreeChunk>> chunksUnknown = chunk_maker.formGroupedPhrasesFromChunksForPara(unknown);
+		List<List<List<ParseTreeChunk>>> posIntersections = new ArrayList<List<List<ParseTreeChunk>>>(), 
+				negIntersections = new ArrayList<List<List<ParseTreeChunk>>>();
+		List<List<ParseTreeChunk>> intersection = null;
+		for(int iConcept = 0; iConcept<psPos.conceptList.size(); iConcept++){
+			if (psPos.conceptList.get(iConcept).intent!=null && psPos.conceptList.get(iConcept).intent.size()>0){
+				intersection = computeIntersectionWithIntentExtendedByDeduction(psPos, iConcept, chunksUnknown);
+				if (reduceList(intersection).size()>0)
+					posIntersections.add(reduceList(intersection));
+			}
+			if (psNeg.conceptList.get(iConcept).intent!=null && psNeg.conceptList.get(iConcept).intent.size()>0){				
+				intersection = md
+						.matchTwoSentencesGroupedChunksDeterministic(psNeg.conceptList.get(iConcept).intent, chunksUnknown);
+				if (reduceList(intersection).size()>0)
+					negIntersections.add(reduceList(intersection));
+			}
+		}
+
+		Pair<List<List<List<ParseTreeChunk>>>, List<List<List<ParseTreeChunk>>>> pair = 
+				removeInconsistenciesFromPosNegIntersections( posIntersections, 
+						negIntersections);
+
+		posIntersections = pair.getFirst();
+		negIntersections = pair.getSecond();
+
+		List<List<List<ParseTreeChunk>>> posIntersectionsUnderNeg = new ArrayList<List<List<ParseTreeChunk>>>(), 
+				negIntersectionsUnderPos = new ArrayList<List<List<ParseTreeChunk>>>();
+
+		for(int iConcept = 0; iConcept<psNeg.conceptList.size(); iConcept++){
+			for(int iConceptJ = 0; iConceptJ<negIntersections.size(); iConceptJ++){
+				intersection = md
+						.matchTwoSentencesGroupedChunksDeterministic(psNeg.conceptList.get(iConcept).intent, negIntersections.get(iConceptJ));
+				if (reduceList(intersection).size()>0)
+					posIntersectionsUnderNeg.add(reduceList(intersection));
+			}
+		}
+
+		for(int iConcept = 0; iConcept<psPos.conceptList.size(); iConcept++){
+			for(int iConceptJ = 0; iConceptJ<posIntersections.size(); iConceptJ++){
+				intersection = md
+						.matchTwoSentencesGroupedChunksDeterministic(psPos.conceptList.get(iConcept).intent, posIntersections.get(iConceptJ));
+				if (reduceList(intersection).size()>0)
+					negIntersectionsUnderPos.add(reduceList(intersection));
+			}
+		}
+
+		List<ParseTreeChunk>posIntersectionsUnderNegLst = flattenParseTreeChunkLst(posIntersectionsUnderNeg);
+		List<ParseTreeChunk>negIntersectionsUnderPosLst=flattenParseTreeChunkLst(negIntersectionsUnderPos);
+
+		posIntersectionsUnderNegLst = subtract(posIntersectionsUnderNegLst, negIntersectionsUnderPosLst);
+		negIntersectionsUnderPosLst= subtract(negIntersectionsUnderPosLst, posIntersectionsUnderNegLst);
+
+		System.out.println("Pos - neg inters = "+posIntersectionsUnderNegLst);
+		System.out.println("Neg - pos inters = "+negIntersectionsUnderPosLst);
+
+		Boolean bPositiveClass = (float)posIntersectionsUnderNegLst.size()/(float)negIntersectionsUnderPosLst.size() > 1f;
+
+		JSMDecision decision = new JSMDecision("keywordClassName", bPositiveClass, 
+				posIntersections , negIntersections, 
+				posIntersectionsUnderNeg,
+				negIntersectionsUnderPos, separationKeywords);
+
+
+		return decision;
+
+	}
+
+	private List<List<ParseTreeChunk>> computeIntersectionWithIntentExtendedByDeduction(
+			LinguisticPatternStructure psPos, int iConcept,
+			List<List<ParseTreeChunk>> chunksUnknown) {
+		
+		 return md
+			.matchTwoSentencesGroupedChunksDeterministic(psPos.conceptList.get(iConcept).intent, chunksUnknown);
+		
+	}
+
+	public Pair<List<String>, List<String>>  reGroupByOccurrenceOfSeparationKeyword(List<String> posTexts, List<String> negTexts, String[] keywords){
+		// do nothing in base class
+
+		return new Pair<List<String>, List<String>>(posTexts, negTexts);
+	}
+
+	public List<List<ParseTreeChunk>> reduceList(List<List<ParseTreeChunk>> list){
+		float minScore = 1.3f;
+		List<List<ParseTreeChunk>> newList = new ArrayList<List<ParseTreeChunk>>();
+
+
+		ParseTreeChunkListScorer scorer = new ParseTreeChunkListScorer();
+		for(  List<ParseTreeChunk> group: list){
+			List<ParseTreeChunk> newGroup = new ArrayList<ParseTreeChunk>();
+			for(ParseTreeChunk ch: group){
+				if (scorer.getScore(ch) > minScore)
+					newGroup.add(ch);
+			}
+			if (newGroup.size()>0)
+				newList.add(newGroup);
+		} 
+
+		return newList;
+
+	}
+
+	public List<List<ParseTreeChunk>> flattenParseTreeChunkListList(List<List<List<ParseTreeChunk>>> listOfLists){
+		List<List<ParseTreeChunk>> newList = new ArrayList<List<ParseTreeChunk>>();
+
+		for(  List<List<ParseTreeChunk>> member: listOfLists){
+			Set<ParseTreeChunk> newSet= new HashSet<ParseTreeChunk>();
+			for(  List<ParseTreeChunk> group: member){
+				if (group.size()>0)
+					newSet.addAll(group);
+			}
+			newList.add(new ArrayList<ParseTreeChunk>(newSet));
+		}
+
+		return newList;  
+	}
+
+	public List<ParseTreeChunk> flattenParseTreeChunkLst(List<List<List<ParseTreeChunk>>> listOfLists){
+		List<ParseTreeChunk> newList = new ArrayList<ParseTreeChunk>();
+		Set<ParseTreeChunk> newSetAll = new HashSet<ParseTreeChunk>();
+
+
+		for(  List<List<ParseTreeChunk>> member: listOfLists){
+			Set<ParseTreeChunk> newSet= new HashSet<ParseTreeChunk>();
+			for(  List<ParseTreeChunk> group: member){
+				if (group.size()>0)
+					newSet.addAll(group);
+			}
+			newSetAll.addAll(newSet);
+		}
+
+		return removeDuplicates(new ArrayList<ParseTreeChunk>(newSetAll));  
+	}
+
+	public List<ParseTreeChunk> removeDuplicates(List<ParseTreeChunk> dupes){
+		List<Integer> toDelete = new ArrayList<Integer>();
+		for(int i=0; i<dupes.size(); i++)
+			for(int j=i+1; j<dupes.size(); j++){
+				if (dupes.get(i).equals(dupes.get(j))){
+					toDelete.add(j);
+				}
+			}
+		List<ParseTreeChunk> cleaned = new ArrayList<ParseTreeChunk>();
+		for(int i=0; i<dupes.size(); i++){
+			if (!toDelete.contains(i))
+				cleaned.add(dupes.get(i));
+		}
+		return cleaned;
+	}
+
+	public List<ParseTreeChunk> subtract(List<ParseTreeChunk> main, List<ParseTreeChunk> toSubtract){
+		List<Integer> toDelete = new ArrayList<Integer>();
+		for(int i=0; i<main.size(); i++)
+			for(int j=0; j<toSubtract.size(); j++){
+				if (main.get(i).equals(toSubtract.get(j))){
+					toDelete.add(i);
+				}
+			}
+		List<ParseTreeChunk> cleaned = new ArrayList<ParseTreeChunk>();
+		for(int i=0; i<main.size(); i++){
+			if (!toDelete.contains(i))
+				cleaned.add(main.get(i));
+		}
+		return cleaned;
+	}
+	public List<ParseTreeChunk> intesectParseTreeChunkLists(List<ParseTreeChunk> a, List<ParseTreeChunk> b){
+		List<Integer> inters = new ArrayList<Integer>();
+		for(int i=0; i<a.size(); i++)
+			for(int j=0; j<b.size(); j++){
+				if (a.get(i).equals(b.get(j))){
+					inters.add(i);
+				}
+			}
+		List<ParseTreeChunk> cleaned = new ArrayList<ParseTreeChunk>();
+		for(int i=0; i<a.size(); i++){
+			if (inters.contains(i))
+				cleaned.add(a.get(i));
+		}
+		return cleaned;
+	}
+
+	public Pair<List<List<List<ParseTreeChunk>>>, List<List<List<ParseTreeChunk>>>>
+	removeInconsistenciesFromPosNegIntersections(List<List<List<ParseTreeChunk>>> pos, 
+			List<List<List<ParseTreeChunk>>> neg ){
+
+		List<ParseTreeChunk> posIntersectionsFl = flattenParseTreeChunkLst(pos);
+		List<ParseTreeChunk> negIntersectionsFl = flattenParseTreeChunkLst(neg);
+
+		List<ParseTreeChunk> intersParseTreeChunkLists = intesectParseTreeChunkLists(posIntersectionsFl, negIntersectionsFl);
+
+		List<List<List<ParseTreeChunk>>> cleanedFromInconsPos = new ArrayList<List<List<ParseTreeChunk>>>(), 
+				cleanedFromInconsNeg = new ArrayList<List<List<ParseTreeChunk>>>();
+		/*
+		System.out.println("pos = "+ pos);
+		System.out.println("neg = "+ neg);
+		System.out.println("pos flat = "+ posIntersectionsFl);
+		System.out.println("neg flat = "+ negIntersectionsFl);
+		System.out.println("inters = "+  intersParseTreeChunkLists);
+		 */
+
+		for(  List<List<ParseTreeChunk>> member: pos){
+			List<List<ParseTreeChunk>> memberList = new ArrayList<List<ParseTreeChunk>>();
+			for( List<ParseTreeChunk> group: member){
+				List<ParseTreeChunk> newGroup = new ArrayList<ParseTreeChunk>();
+				for(ParseTreeChunk ch: group){
+					boolean bSkip = false;	 
+					for(ParseTreeChunk check: intersParseTreeChunkLists){
+						if (check.equals(ch))
+							bSkip=true;
+					}
+					if (!bSkip)
+						newGroup.add(ch);
+				}
+				if (newGroup.size()>0)
+					memberList.add(newGroup);
+			} 
+			if (memberList.size()>0)
+				cleanedFromInconsPos.add(memberList);
+		}
+
+		for(  List<List<ParseTreeChunk>> member: neg){
+			List<List<ParseTreeChunk>> memberList = new ArrayList<List<ParseTreeChunk>>();
+			for( List<ParseTreeChunk> group: member){
+				List<ParseTreeChunk> newGroup = new ArrayList<ParseTreeChunk>();
+				for(ParseTreeChunk ch: group){
+					boolean bSkip = false;	 
+					for(ParseTreeChunk check: intersParseTreeChunkLists){
+						if (check.equals(ch))
+							bSkip=true;
+					}
+					if (!bSkip)
+						newGroup.add(ch);
+				}
+				if (newGroup.size()>0)
+					memberList.add(newGroup);
+			} 
+			if (memberList.size()>0)
+				cleanedFromInconsNeg.add(memberList);
+		}
+		return  new Pair<List<List<List<ParseTreeChunk>>>, List<List<List<ParseTreeChunk>>>>(cleanedFromInconsPos, cleanedFromInconsNeg);
+	}
+
+
+	public static void main (String[] args) {
+
+		String[] posArr = new String[] {"I rent an office space. This office is for my business. I can deduct office rental expense from my business profit to calculate net income. ",
+				"To run my business, I have to rent an office. The net business profit is calculated as follows. Rental expense needs to be subtracted from revenue. ",
+				"To store goods for my retail business I rent some space. When I calculate the net income, I take revenue and subtract business expenses such as office rent. ",
+		"I rent some space for my business. To calculate my net income, I subtract from revenue my rental business expense."};
+
+		String[] negArr = new String[] {"I rent out a first floor unit of my house to a travel business. I need to add the rental income to my profit. However, when I repair my house, I can deduct the repair expense from my rental income. ",
+				"I receive rental income from my office. I have to claim it as a profit in my tax forms. I need to add my rental income to my profits, but subtract rental expenses such as repair from it. ",
+				"I advertised my property as a business rental. Advertisement and repair expenses can be subtracted from the rental income. Remaining rental income needs to be added to my profit and be reported as taxable profit. ",			
+		"I showed  my property to a business owner to rent. Expenses on my time spent on advertisement are subtracted from the rental income. My rental profits are added to my taxable income.  "};	
+
+		String unknown = "I do not want to rent anything to anyone. I just want to rent a space for myself. I neither calculate deduction of individual or business tax. I subtract my tax from my income";
+
+		JSMDecision dec = new JSMLearnerOnLatticeBase().
+				buildLearningModel(Arrays.asList(posArr), Arrays.asList(negArr), unknown, null);
+
+
+	}
+}

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/main/java/opennlp/tools/jsmlearning/JSMLearnerOnLatticeWithAbduction.java
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/main/java/opennlp/tools/jsmlearning/JSMLearnerOnLatticeWithAbduction.java b/opennlp-similarity/src/main/java/opennlp/tools/jsmlearning/JSMLearnerOnLatticeWithAbduction.java
new file mode 100644
index 0000000..501e908
--- /dev/null
+++ b/opennlp-similarity/src/main/java/opennlp/tools/jsmlearning/JSMLearnerOnLatticeWithAbduction.java
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package opennlp.tools.jsmlearning;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.commons.collections.ListUtils;
+
+import edu.stanford.nlp.util.StringUtils;
+import opennlp.tools.fca.ConceptLattice;
+import opennlp.tools.fca.FcaWriter;
+import opennlp.tools.fca.FormalConcept;
+import opennlp.tools.parse_thicket.pattern_structure.LinguisticPatternStructure;
+import opennlp.tools.similarity.apps.BingWebQueryRunner;
+import opennlp.tools.similarity.apps.HitBase;
+import opennlp.tools.similarity.apps.utils.Pair;
+import opennlp.tools.textsimilarity.ParseTreeChunk;
+import opennlp.tools.textsimilarity.ParseTreeChunkListScorer;
+import opennlp.tools.textsimilarity.ParseTreeMatcherDeterministic;
+import opennlp.tools.textsimilarity.chunker2matcher.ParserChunker2MatcherProcessor;
+
+public class JSMLearnerOnLatticeWithAbduction extends JSMLearnerOnLatticeWithDeduction{
+
+
+
+
+	public JSMDecision buildLearningModel(List<String> posTexts, List<String> negTexts, 
+			String unknown, String[] separationKeywords){
+		JSMDecision decision = super.buildLearningModel(posTexts, negTexts, unknown, separationKeywords);
+		// verify each hypothesis
+		//TODO
+				return decision;
+
+	}
+
+
+	
+
+	public static void main (String[] args) {
+
+		String[] posArr = new String[] {"I rent an office space. This office is for my business. I can deduct office rental expense from my business profit to calculate net income. ",
+				"To run my business, I have to rent an office. The net business profit is calculated as follows. Rental expense needs to be subtracted from revenue. ",
+				"To store goods for my retail business I rent some space. When I calculate the net income, I take revenue and subtract business expenses such as office rent. ",
+		"I rent some space for my business. To calculate my net income, I subtract from revenue my rental business expense."};
+
+		String[] negArr = new String[] {"I rent out a first floor unit of my house to a travel business. I need to add the rental income to my profit. However, when I repair my house, I can deduct the repair expense from my rental income. ",
+				"I receive rental income from my office. I have to claim it as a profit in my tax forms. I need to add my rental income to my profits, but subtract rental expenses such as repair from it. ",
+				"I advertised my property as a business rental. Advertisement and repair expenses can be subtracted from the rental income. Remaining rental income needs to be added to my profit and be reported as taxable profit. ",			
+		"I showed  my property to a business owner to rent. Expenses on my time spent on advertisement are subtracted from the rental income. My rental profits are added to my taxable income.  "};	
+
+		String unknown = "I do not want to rent anything to anyone. I just want to rent a space for myself. I neither calculate deduction of individual or business tax. I subtract my tax from my income";
+		JSMLearnerOnLatticeWithAbduction jsm = new JSMLearnerOnLatticeWithAbduction();
+		JSMDecision dec1 =  // may be determined by 'subtract'
+				jsm.buildLearningModel(Arrays.asList(posArr), Arrays.asList(negArr), unknown , new String[]{"subtract"});
+		JSMDecision dec2 = // may be determined by ...
+				jsm.buildLearningModel(Arrays.asList(posArr), Arrays.asList(negArr), unknown , new String[]{"business"});
+		JSMDecision dec3 = // may be determined by ...
+				jsm.buildLearningModel(Arrays.asList(posArr), Arrays.asList(negArr), unknown , new String[]{"property"});
+		// Finally, do prediction
+		JSMDecision dec = // may be determined by ...
+				jsm.buildLearningModel(Arrays.asList(posArr), Arrays.asList(negArr), unknown , new String[]{"property"});
+		
+		
+		
+
+
+	}
+}


[40/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib2/37Bib2Mark_Gospel_12_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib2/37Bib2Mark_Gospel_12_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib2/37Bib2Mark_Gospel_12_EN.txt.txt
new file mode 100644
index 0000000..9c3c577
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Bib2/37Bib2Mark_Gospel_12_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Mark 1 John the Baptist Prepares the Way 1 The beginning of the gospel about Jesus Christ , the Son of God . [ a ] 2 It is written in Isaiah the prophet : \u201c I will send my messenger ahead of you , who will prepare your way\u201d[b ] \u2014 3 \u201c a voice of one calling in the desert , \u2018 Prepare the way for the Lord , make straight paths for him . '\u201d[c ] 4 And so John came , baptizing in the desert region and preaching a baptism of repentance for the forgiveness of sins . 5 The whole Judean countryside and all the people of Jerusalem went out to him . Confessing their sins , they were baptized by him in the Jordan River . 6 John wore clothing made of camel 's hair , with a leather belt around his waist , and he ate locusts and wild honey . 7 And this was his message : \u201c After me will come one more powerful than I , the thongs of whose sandals I am not worthy to stoop down and untie . 8 I baptize you with[d ] water , but he will baptize you with the Holy Spirit . \u201d The Baptism 
 and Temptation of Jesus 9 At that time Jesus came from Nazareth in Galilee and was baptized by John in the Jordan . 10 As Jesus was coming up out of the water , he saw heaven being torn open and the Spirit descending on him like a dove . 11 And a voice came from heaven : \u201c You are my Son , whom I love ; with you I am well pleased . \u201d 12 At once the Spirit sent him out into the desert , 13 and he was in the desert forty days , being tempted by Satan . He was with the wild animals , and angels attended him . The Calling of the First Disciples 14 After John was put in prison , Jesus went into Galilee , proclaiming the good news of God . 15 \u201c The time has come , \u201d he said . \u201c The kingdom of God is near . Repent and believe the good news ! \u201d 16 As Jesus walked beside the Sea of Galilee , he saw Simon and his brother Andrew casting a net into the lake , for they were fishermen . 17 \u201c Come , follow me , \u201d Jesus said , \u201c and I will make you fishers of men . \u201d 18 At once 
 they left their nets and followed him . 19 When he had gone a little farther , he saw James son of Zebedee and his brother John in a boat , preparing their nets . 20 Without delay he called them , and they left their father Zebedee in the boat with the hired men and followed him . Jesus Drives Out an Evil Spirit 21 They went to Capernaum , and when the Sabbath came , Jesus went into the synagogue and began to teach . 22 The people were amazed at his teaching , because he taught them as one who had authority , not as the teachers of the law . 23 Just then a man in their synagogue who was possessed by an evil[e ] spirit cried out , 24 \u201c What do you want with us , Jesus of Nazareth ? Have you come to destroy us ? I know who you are\u2014the Holy One of God ! \u201d 25 \u201c Be quiet ! \u201d said Jesus sternly . \u201c Come out of him ! \u201d 26 The evil spirit shook the man violently and came out of him with a shriek . 27 The people were all so amazed that they asked each other , \u201c What is this ?
  A new teaching\u2014and with authority ! He even gives orders to evil spirits and they obey him . \u201d 28 News about him spread quickly over the whole region of Galilee . Jesus Heals Many 29 As soon as they left the synagogue , they went with James and John to the home of Simon and Andrew . 30 Simon 's mother-in-law was in bed with a fever , and they told Jesus about her . 31 So he went to her , took her hand and helped her up . The fever left her and she began to wait on them . 32 That evening after sunset the people brought to Jesus all the sick and demon-possessed . 33 The whole town gathered at the door , 34 and Jesus healed many who had various diseases . He also drove out many demons , but he would not let the demons speak because they knew who he was . Jesus Prays in a Solitary Place 35 Very early in the morning , while it was still dark , Jesus got up , left the house and went off to a solitary place , where he prayed . 36 Simon and his companions went to look for him , 37 and 
 when they found him , they exclaimed : \u201c Everyone is looking for you ! \u201d 38 Jesus replied , \u201c Let us go somewhere else\u2014to the nearby villages\u2014so I can preach there also . That is why I have come . \u201d 39 So he traveled throughout Galilee , preaching in their synagogues and driving out demons . A Man With Leprosy 40 A man with leprosy[f ] came to him and begged him on his knees , \u201c If you are willing , you can make me clean . \u201d 41 Filled with compassion , Jesus reached out his hand and touched the man . \u201c I am willing , \u201d he said . \u201c Be clean ! \u201d 42 Immediately the leprosy left him and he was cured . 43 Jesus sent him away at once with a strong warning : 44 \u201c See that you do n't tell this to anyone . But go , show yourself to the priest and offer the sacrifices that Moses commanded for your cleansing , as a testimony to them . \u201d 45 Instead he went out and began to talk freely , spreading the news . As a result , Jesus could no longer enter a town openly but 
 stayed outside in lonely places . Yet the people still came to him from everywhere . Mark 2 Jesus Heals a Paralytic 1 A few days later , when Jesus again entered Capernaum , the people heard that he had come home . 2 So many gathered that there was no room left , not even outside the door , and he preached the word to them . 3 Some men came , bringing to him a paralytic , carried by four of them . 4 Since they could not get him to Jesus because of the crowd , they made an opening in the roof above Jesus and , after digging through it , lowered the mat the paralyzed man was lying on . 5 When Jesus saw their faith , he said to the paralytic , \u201c Son , your sins are forgiven . \u201d 6 Now some teachers of the law were sitting there , thinking to themselves , 7 \u201c Why does this fellow talk like that ? He 's blaspheming ! Who can forgive sins but God alone ? \u201d 8 Immediately Jesus knew in his spirit that this was what they were thinking in their hearts , and he said to them , \u201c Why ar
 e you thinking these things ? 9 Which is easier : to say to the paralytic , \u2018 Your sins are forgiven , ' or to say , \u2018 Get up , take your mat and walk ' ? 10 But that you may know that the Son of Man has authority on earth to forgive sins \u2026 \u201d He said to the paralytic , 11 \u201c I tell you , get up , take your mat and go home . \u201d 12 He got up , took his mat and walked out in full view of them all . This amazed everyone and they praised God , saying , \u201c We have never seen anything like this ! \u201d The Calling of Levi 13 Once again Jesus went out beside the lake . A large crowd came to him , and he began to teach them . 14 As he walked along , he saw Levi son of Alphaeus sitting at the tax collector 's booth . \u201c Follow me , \u201d Jesus told him , and Levi got up and followed him . 15 While Jesus was having dinner at Levi 's house , many tax collectors and \u201c sinners \u201d were eating with him and his disciples , for there were many who followed him . 16 When the teachers of the
  law who were Pharisees saw him eating with the \u201c sinners \u201d and tax collectors , they asked his disciples : \u201c Why does he eat with tax collectors and \u2018 sinners ' ? \u201d 17 On hearing this , Jesus said to them , \u201c It is not the healthy who need a doctor , but the sick . I have not come to call the righteous , but sinners . \u201d Jesus Questioned About Fasting 18 Now John 's disciples and the Pharisees were fasting . Some people came and asked Jesus , \u201c How is it that John 's disciples and the disciples of the Pharisees are fasting , but yours are not ? \u201d 19 Jesus answered , \u201c How can the guests of the bridegroom fast while he is with them ? They cannot , so long as they have him with them . 20 But the time will come when the bridegroom will be taken from them , and on that day they will fast . 21 \u201c No one sews a patch of unshrunk cloth on an old garment . If he does , the new piece will pull away from the old , making the tear worse . 22 And no one pours new wine into 
 old wineskins . If he does , the wine will burst the skins , and both the wine and the wineskins will be ruined . No , he pours new wine into new wineskins . \u201d Lord of the Sabbath 23 One Sabbath Jesus was going through the grainfields , and as his disciples walked along , they began to pick some heads of grain . 24 The Pharisees said to him , \u201c Look , why are they doing what is unlawful on the Sabbath ? \u201d 25 He answered , \u201c Have you never read what David did when he and his companions were hungry and in need ? 26 In the days of Abiathar the high priest , he entered the house of God and ate the consecrated bread , which is lawful only for priests to eat . And he also gave some to his companions . \u201d 27 Then he said to them , \u201c The Sabbath was made for man , not man for the Sabbath . 28 So the Son of Man is Lord even of the Sabbath . \u201d 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Corp/38CorpApple_Environment_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Corp/38CorpApple_Environment_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Corp/38CorpApple_Environment_EN.txt.txt
new file mode 100644
index 0000000..81255cb
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Corp/38CorpApple_Environment_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Apple and the Environment Frequently Asked Questions Q : How does Apple review the company\u2019s progress and status regarding its environmental impact ? A : Apple\u2019s Executive Team is responsible for setting and implementing environmental policies . This group of the company\u2019s most senior executives is advised by teams from across Apple , including environmental engineering , product development , operations , facilities , and supplier responsibility . Apple\u2019s climate change strategy is built into all facilities development plans and plays an important role in the product development process . In 2009 , at the direction of the Executive Team , Apple became the first company in the electronics industry to report its entire carbon footprint including the impact of its products on the environment through consumer use . In addition to providing consumers with an unprecedented level of detail and transparency on which to base their purchasing decisions , measuring the company\u2019s
  carbon footprint helps the company identify areas where it can make the most significant reductions . We understand that the vast majority of greenhouse gas emissions ( up to 98 percent ) come from the product life cycle , which is why decisions about the environmental attributes of our products are an integral aspect of all decisions related to the design and manufacture of those products . The same is true of all the environmental decisions for our facilities . Our integrated approach means that decisions about environmental issues are reviewed at the highest levels of the company . This structure has been an important factor in Apple\u2019s success , particularly in the area of environmental responsibility . Executive Team members regularly review each new product during its development , focusing on material and design choices , the supply chain , packaging , and product energy efficiency . Each of these areas has a direct impact on our environmental footprint . This strategy , wh
 ich we\u2019ve been using for well over two decades , has yielded the most environmentally forward-looking product designs in our industry . Every product we offer meets the ENERGY STAR guidelines for energy efficiency , is free of harmful toxins , and is made of highly recyclable materials . Our facilities and operations teams also regularly review our programs and investments to reduce carbon emissions with Executive Team members . The results of these reviews include increased use of green power , new technology investments , commute programs , and conservation . The efforts of this team have produced the highest product recycling rate in our industry . Apple delivers important environmental attributes across all the company\u2019s products and product families . Not only does Apple deliver where others have only promised to do so in the future , Apple has delivered where it\u2019s most meaningful . The Apple Executive Team is committed to our environmental strategy : to reduce our carbon
  footprint , to remove toxins , and to report on the environmental impact of every product we make so that our customers can measure our progress . We are also committed to reporting our global corporate emissions annually and to reporting at the product level with each new product introduction . Q : What is Life Cycle Assessment ( LCA ) ? A : LCA is a process of evaluating the effects that a product has on the environment throughout its life cycle . The objective of completing a full life cycle analysis is to improve resource efficiency while reducing the environmental impact of all activities required to manufacture , distribute , use , and dispose of a product . Cradle-to-grave life cycle assessments , for instance , begin with the extraction of raw materials from the earth , their transformation into finished materials , the manufacturing of materials into parts and products , transportation to end users , the energy consumed during use , and ultimately recycling . During each o
 f these phases , activities occur that use materials and energy , generating waste and emissions . Life cycle assessment is the process used to accurately measure the emissions associated with each phase of these activities . LCA results are often categorized in several ways , including global warming potential , acidification , eutrophication , tropospheric ozone creation , and stratospheric ozone depletion . The procedures followed for life cycle assessment are defined as part of the internationally recognized ISO 14000 environmental management standards : in ISO 14040:2006 and 14044:2006 . Successful implementations of LCAs have been achieved in many industries , including automobile , airline , and consumer goods companies . For complex products and processes composed of several thousand materials , the analysis requires dedicated software applications designed to accurately capture the environmental impact of very specific processes . Software tools provide an efficient and obj
 ective assessment of the environmental impacts a company measures by using third-party databases created and updated by independent industrial engineers who are experts in specific manufacturing processes . Apple has found that a combination of data collected directly from our internal and supplier processes as well as independent third-party data sets achieves the most accurate results . Q : How does Apple use LCA ? A : Apple uses five steps when conducting a product LCA : 1. Data collection starts with measuring the power consumed by a product while it is running under a simulated use scenario . Different scenarios are used for different product lines . For example , we assume a four-year average use period for Mac computers . To be conservative , we also assume an office scenario use pattern for every computer shipped and apply that use pattern to 365 days of use per year . For handheld products , such as iPod and iPhone , we assume a three-year use period . For these products , 
 charging patterns are defined according to a \u201c power user , \u201d which we define as intense daily use over the three-year period . More information on our product power modes is provided in our Product Environmental Reports . 2. The second step is a part-by-part measurement of the entire product . This helps us accurately measure the size and weight of the components and materials in the product . Since a single product can contain several hundred parts , this step is supported by double-checking individual part details with the product\u2019s bill of materials . We include material and component yield loss during production as part of this process . We also account for the transportation of materials between manufacturing sites . Production emissions for external accessories , such as keyboards and mice , and packaging are also included in this step . 3. The third step assesses emissions from transportation of finished products to sales regions . Data is collected on single units and
  multipack units of product shipments as they are transported over land , by sea , and by air . The majority of the product transportation emissions are associated with getting the products from Asia to sales distribution hubs in Europe , Asia , and the Americas . The final leg of transport between regional hubs and individual customers is accounted for using an overall adjustment factor . 4. The last step of the product life cycle measurement is the collection and recycling of the product when it reaches the end of its useful life . Transportation between customers and recycling plants along with the treatment steps carried out by the recycler to obtain metal , plastic , and glass material streams are all accounted for in this calculation . Subsequent processing and smelting steps are not included as these are considered stages of production and not end-of-life processing . 5. After we collect all the data , the final step is to run the product data model in our LCA tool and compil
 e detailed results for GHG emissions as they relate to the product . The data and life cycle model used in the tool are checked for quality and accuracy by the Fraunhofer Institute in Germany . Q : Does Apple index its environmental performance data using Global Reporting Initiative ( GRI ) Sustainability Reporting Guidelines ( G3 ) ? A : Yes . Environmental performance data covering energy and water consumption , greenhouse gas emissions , and waste production is provided in the 2009 Facilities Report ( PDF ) . Q : What is ISO 14001 , and is Apple ISO 14001 certified ? A : ISO 14001 is a voluntary international standard that establishes the requirements for an organization\u2019s environmental management system ( EMS ) . The ISO 14001 standard helps a company manage the environmental impact of its operations in a systematic way that is integrated with overall business management processes . Apple first achieved ISO 14001 certification for a manufacturing site in 1996. Q : Does Apple b
 an environmentally sensitive substances ? A : Yes . Apple\u2019s Regulated Substances Specification details a broad range of substances that are restricted or banned from use in Apple products , packaging , and manufacturing . Q : Do Apple products contain lead ? A : Apple is in compliance with the European RoHS Directive , which restricts the use of lead and other substances . As a result of our precautionary approach to substances , Apple phased out lead in plastic parts , paint , and packaging material long before the RoHS Directive came into effect . Q : What are Apple\u2019s restrictions on brominated flame retardant-free ( BFR-free ) and polyvinyl chloride-free ( PVC-free ) products ? A : Apple defines a material as BFR-free and PVC-free if it contains less than 900 parts per million ( ppm ) of bromine and chlorine . The 900-ppm limit is one of the strictest in the electronics industry and a concentration lower than the ban on lead in the European RoHS Directive . Since BFRs and PVC
  need to be present in much higher quantities to be effective , most materials that contain less than 900-ppm bromine and chlorine do not have any detectable levels of bromine or chlorine present . Q : Which products are covered by Apple\u2019s BFR-free and PVC-free transition ? A : Apple\u2019s phaseout of BFRs and PVC covers all parts of new Apple product designs manufactured after December 31 , 2008. The iPod nano , iPod classic , and iPod touch are examples of products that are BFR-free and PVC-free . While Apple\u2019s phaseout will cover the vast majority of products and components , older product designs , replacement parts , and accessories for older product designs may not be fully BFR-free and PVC-free . Q : What is REACH , and how is Apple complying with the REACH regulation ? A : The Registration , Evaluation , Authorization and Restriction of Chemicals Regulation EC 1907/2007 , commonly referred to as REACH , is a European regulation on chemicals and their safe use . With the pu
 blication of the candidate list for authorization on October 28 , 2008 , the European Chemicals Agency identified a set of substances of very high concern ( SVHC ) that manufacturers must disclose to customers if used in their products above 0.1 percent . Based on the current version of the candidate list for authorization , Apple products do not contain SVHC above 0.1 percent , with the exception of AC power cords . AC power cords used in Apple notebooks , desktops , servers , displays , Apple TV , AirPort Extreme , Time Capsule , and power adapters are made of polyvinyl chloride ( PVC ) that contains the SVHC bis 2-ethyl(hexyl)phthalate , also known as DEHP . Apple is in the final stages of developing and certifying PVC-free AC power cords that will also be free of DEHP . Q : Does Apple use ozone-depleting substances ? A : No ozone-depleting substances ( ODCs ) are used in any processes to manufacture components , materials , or product packaging materials used by Apple , as stipu
 lated by the Montreal Protocol on Substances That Deplete the Ozone Layer . Q : Does Apple have an Environmental Health and Safety Policy ? A : Apple is committed to protecting the environment , health , and safety of our employees , customers , and the global communities in which we operate . For more information , read our Environmental Health and Safety Policy statement ( PDF ) . Q : What happens to computers when they are recycled ? A : When you recycle with Apple , your used equipment is disassembled , and key components that can be reused are removed . Glass and metal can be reprocessed for use in new products . A majority of the plastics can be pelletized into a raw secondary material . With materials reprocessing and component reuse , Apple often achieves a 90 percent recovery rate by weight of the original product . Learn more about the Apple Recycling Program 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Corp/39CorpHandM_Quality_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Corp/39CorpHandM_Quality_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Corp/39CorpHandM_Quality_EN.txt.txt
new file mode 100644
index 0000000..587eaa5
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Corp/39CorpHandM_Quality_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff FAQ Will my H&M garments shrink ? H&M accept a shrinkage of up to 3 % for woven garments and 5 % for knits/ jerseys . To avoid shrinkage , certain items might need ironing after washes , to regain their original shape and fit . If needed , this will always be mentioned on the care label . Are hazardous or allergenic substances being used in your production methods ? H&M ban the use of hazardous and environmentally damaging substances thoughout all production processes . All H&M suppliers commit to comply with our requirements and are educated in how to run production to our standards . Regular controls and inspections are also conducted to ensure your safety . What do you do with defective and surplus garments ? Clothes that do not meet H&M\u2019s quality requirements are not passed on to the stores . Instead , such garments are donated to organizations like UNHCR , the Red Cross and Caritas . We do not give away garments that do not comply with our safety requirements and chemical
  restrictions , or that are damaged . When feasible , our stores donate defective but usable garments that have been returned by our customers . We are reviewing our routines to see whether there is scope for further improvement . Do I need to wash my H&M clothes before wearing them for the first time ? As hazardous chemicals are banned thoughout our production , washing before wearing is not neccessary . Is my ski jacket or overall from H&M waterproof ? H&M has three levels of functional garments with water repellent properties . Information will be found on the label . Waterproof : The fabric is waterproof and has sealed seams . Suitable for heavy rain conditions . Water resistant : The fabric is waterproof and has folded seams . Suitable for rainy or snowy conditions . Water repellent : The fabric 's surface has been treated to repel water . Suitable for light rain . Can I wash at a lower temperature than recommended on the care label ? The highest recommended washing temperature
  is stated on the garment 's label . You can , of course , choose a lower temperature to save energy . However , heavily soiled clothes might require the higher temperature . Can wire bras be machine washed ? We always recommend hand washing for wire bras , as machine washing could cause the wire to separate and damage the machine . Do H&M sunglasses come with UV-protection ? Yes , all H&M sunglasses have a special UV filter which locks out all UV radiations ( UVA and UVB ) . How do I wash knitted wool garments ? Make sure you follow the care instructions on the label . If your washing machine has a wool program , use this . If hand washed , remember to carefully squeeze any extra water out of the garment . Reshape the garment after wash and dry flat for optimal care . How do I choose my right size at H&M ? An easy way to calculate your size is to use our Size Guide , found here . Please note that same size garments may still vary depending on fit . H&M follow the standard body meas
 urement lists , based on average body measurements for all sizes . Do your accessories contain nickel metals ? H&M abide by the recommended limitation of nickel metal use in all products . We also conduct regular tests to make sure the limit is not exceeded . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Corp/40CorpHandM_Responsibility_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Corp/40CorpHandM_Responsibility_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Corp/40CorpHandM_Responsibility_EN.txt.txt
new file mode 100644
index 0000000..9d1120a
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Corp/40CorpHandM_Responsibility_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff FAQ How does H&M check that suppliers abide by the Code of Conduct ? We employ full-time auditors whose job it is to check compliance with our Code of Conduct . During audits they go through a list containing over 300 points , relating to working conditions , the working environment , etc. After each audit we collate the results in a report that specifies the areas where improvements are needed , and the supplier is set a deadline for submitting an action plan . The auditors then visit the factories to follow up whether these actions have been taken . H&M also supports suppliers \u2019 improvement work through training and projects of various kinds . Read more about our Code of Conduct follow-up here . Is it possible to buy clothes produced with limited environmental impact at a good price ? H&M\u2019s business concept is to offer fashion and quality at the best price . We manage this by buying direct from the manufacturers , by having efficient logistics and by having our own stores 
 \u2013 not by compromising on our environmental requirements . We are also cost-conscious at every stage and the fact is that a cheaper mode of transport \u2013 such as by ship \u2013 causes lower carbon dioxide emissions than more expensive means of transport , such as aircraft . So there is no conflict of interest between good prices and low climate impact . We make efforts to support our suppliers so that production takes place with consideration for the health of customers and workers as well as for the environment . What is H&M 's policy regarding fur and leather products ? H&M does not sell real fur . H&M only sells leather from sheep , pigs , goats and cattle that have been bred for meat production , not just for their skin . No other leather is permitted in products sold by H&M . Are H&M cosmetics tested on animals ? No animal testing is carried out on our cosmetics products , neither during production nor on the finished products . Why do H&M not use Fairtrade cerified cotton ? H&M 
 constantly works to improve conditions in cotton farming . One of the ways we do so , is through our active involvement in the Better Cotton Initiative ( BCI ) , where we are members of the steering comittee . BCI 's vision is to enable millions of farmers around the world to grow cotton in a way thats more beneficial for both the farming community and the environment . Read more about BCI at www . bettercotton . org . H&M also offers clothes made with organic cotton . The cotton used in these clothes is 100 % organically grown cotton , certified by the Control Union . We also incorporate organic cotton into some of our children\u2019s and babies \u2019 wear . We hope that we can thereby show cotton farmers that there is a demand for organic cotton and encourage them to switch from conventional to organic production . Read more about the certification of organic cotton at www . controlunion . com . How can H&M ensure that your products are free from harmful chemicals ? The health and safe
 ty of our customers is a very high priority concern , which is why we have been actively working for more than ten years on restricting the use of chemicals that may be harmful to both your health and the environment . We do this by means of our Chemical Restrictions \u2013 a list of restrictions that all suppliers must undertake to observe . We regularly test our products to ensure that they do not contain harmful chemicals . We apply the precautionary principle , which means that we do not only ban the use of substances that have been proven to be harmful , but also those merely suspected of being so . This completely ensures that our customers can feel safe with our products . Why does H&M have their own auditors ? Having our own auditors makes it easier to work for long-term improvements in our suppliers \u2019 factories . Moreover , integrating CSR work into our day-to-day operations takes very high priority at H&M . When choosing suppliers , for example , compliance with our Code of
  Conduct is one of the aspects we consider . This work is made considerably easier by having our own audit team at the offices in the countries that handle production . Monitoring and improvement work in the factories is the H&M auditors sole task . They are not required to consider commercial aspects , but instead make independent decisions that our buying organisation must respect . To complement our own monitoring programme , H&M participates in the Fair Labor Association ( FLA ) . Among other things , this participation means that the FLA makes unannounced independent external monitoring visits to a number of H&M\u2019s suppliers ' factories in China and Turkey . We hope that this will show our customers and other stakeholders how well our follow-up in the factories is working . In addition , participating in the FLA provides good opportunities to cooperate with other companies and the FLA\u2019s partner organisations , with the aim of improving working standards in our supply chain .
  We also use the results of the FLA\u2019s audits as a point of reference for quality assurance of our internal monitoring programme . How can you guarantee merino wool products are mulesing-free ? H&M considers it important that animals are not mistreated . We have therefore distanced ourselves from mulesing and concentrated our buying on suppliers that can guarantee mulesing-free merino wool . How do you ensure that child labour is n't used during production ? H&M\u2019s auditors carry out regular checks to ensure that there are no underage workers in the factories . It is extremely rare for child labour to be discovered at H&M 's suppliers or their subcontractors . Should this nonetheless happen , H&M requires the supplier to take responsibility and , together with H&M and the family , to find a solution in the best interests of the child . An investigation is then carried out , looking at how best to resolve the situation based on the child\u2019s interests . On many occasions , the solu
 tion will involve the supplier contributing financially so that the child can receive an education , while compensating the family for the loss of income . If H&M discovers repeated breaches of its ban on child labour at a supplier \u2013 or one of its subcontractors \u2013 the cooperation will be ended for good . How do I know where my H&M items were produced ? All products in H&M\u2019s range have the country of origin stated on the label . Why are the factory workers wages so low ? Since H&M neither owns nor operates the factories that make our clothes , we do not set or pay the factory employees \u2019 wages . We are rarely the suppliers \u2019 only customer and a worker may produce items for a number of different buyers in the same month . We believe that all the employees of a factory should have the same earnings potential , regardless of which buyer they are making clothes for . But we demand \u2013 and check \u2013 that everyone receives the pay and overtime remuneration to which they are legal
 ly entitled . That is the only practicable solution \u2013 and the most competition-neutral . Our Code of Conduct states that the statutory minimum wage is the lowest acceptable pay level for our suppliers \u2019 employees , but not the recommended level . We prefer to see pay levels based on negotiations between employer and employees , and for the law to lay down a reasonable minimum wage . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Corp/41CorpHP_Objectives_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Corp/41CorpHP_Objectives_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Corp/41CorpHP_Objectives_EN.txt.txt
new file mode 100644
index 0000000..7f9b755
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Corp/41CorpHP_Objectives_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Corporate Objectives \u201c It is necessary that people work together in unison toward common objectives and avoid working at cross purposes at all levels if the ultimate in efficiency and achievement is to be obtained . \u201d \u2013 Dave Packard HP 's Corporate Objectives have guided the company in the conduct of its business since 1957 , when first written by co-founders Bill Hewlett and Dave Packard . Customer loyalty To provide products , services and solutions of the highest quality and deliver more value to our customers that earns their respect and loyalty . Underlying beliefs supporting this objective : * Our continued success is dependent on increasing the loyalty of our customers . * Listening attentively to customers to truly understands their needs , then delivering solutions that translate into customer success is essential to earn customer loyalty . * Competitive total cost of ownership , quality , inventiveness , and the way we do business drives customer loyalty . Profit
  To achieve sufficient profit to finance our company growth , create value for our shareholders and provide the resources we need to achieve our other corporate objectives . Underlying beliefs supporting this objective : * Profit is the responsibility of all . * Balance of long-term and short-term objectives is key to profitability . * Profit allows us to reinvest in new and emerging business opportunities . * Profit is highly correlated to generating cash , which brings more flexibility to the business at a lower cost . * Profit enables the achievement of our corporate objectives . Market leadership To grow by continually providing useful and significant products , services and solutions to markets we already serve\u2014and to expand into new areas that build on our technologies , competencies and customer interests . Underlying beliefs supporting this objective : * There are more places we can contribute than we will be capable of contributing : We must focus . * To be average in the
  marketplace is not good enough , we play to win . * We must be No. 1 or No. 2 in our chosen fields . Growth To view change in the market as an opportunity to grow ; to use our profits and our ability to develop and produce innovative products , services and solutions that satisfy emerging customer needs . Underlying beliefs supporting this objective : * Growth comes from taking smart risks , based on the state of the industry\u2014that requires both a conviction in studying the trends , but also in inducing change in our industry . * Our size ( and diversity of businesses ) gives us an ability to weather economic cycles and turn them to our favor . Employee commitment To help HP employees share in the company 's success that they make possible ; to provide people with employment opportunities based on performance ; to create with them a safe , exciting and inclusive work environment that values their diversity and recognizes individual contributions ; and to help them gain a sense of 
 satisfaction and accomplishment from their work . Underlying beliefs supporting this objective : * HP 's performance starts with motivated employees ; their loyalty is key . * We trust our employees to do the right thing and to make a difference . * Everyone has something to contribute : It 's not about title , level or tenure . * An exciting , stimulating work environment is critical to invention . * A diverse workforce gives us a competitive advantage . * Employees are responsible for lifelong learning . Leadership capability To develop leaders at every level who are accountable for achieving business results and exemplifying our values . Underlying beliefs supporting this objective : * Leaders inspire , foster collaboration and turn vision and strategies into action\u2014with focused , clear goals . * Effective leaders coach , relay good news and bad , and give feedback that works . * Leaders demonstrate self-awareness and a willingness to accept feedback and continuously develop . 
 * Leaders speak with one voice and act to eliminate busy work . * It is important to measure people on the results they achieve against goals they helped to create . Global citizenship Good citizenship is good business . We live up to our responsibility to society by being an economic , intellectual and social asset to each country and community in which we do business . Underlying beliefs supporting this objective : * The highest standards of honesty and integrity are critical to developing customer and stakeholder loyalty . * The betterment of our society is not a job to be left to a few ; it is the responsibility to be shared by all . * This objective is essential to delivering on the brand promise . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Corp/42CorpJaguar_Sustainability_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Corp/42CorpJaguar_Sustainability_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Corp/42CorpJaguar_Sustainability_EN.txt.txt
new file mode 100644
index 0000000..352b815
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Corp/42CorpJaguar_Sustainability_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff SUSTAINABLE DEVELOPMENT POLICY Jaguar Sustainability Our Policy * We are committed to addressing the challenges of sustainable development and operating as a responsible corporate business . We will strive to achieve sustainability by integrating and balancing our economic , social and environmental responsibilities within all our activities . We will develop and operate efficient business processes that are holistic , encompassing economic , social and environmental requirements , and that are an integral part of the decision-making process . We will implement tools and techniques to enable us to assess our sustainability impacts , to ensure performance improvement , and periodically we will report on our performance . Fundamental to our operation is the commitment to continual improvement , and to meet or exceed legal and regulatory requirements in all of our activities . * ECONOMIC - Deliver financial growth with a focus on a long term sustainability , recognising our respons
 ibilities to the wide range of stakeholders - Account for the true impact of our activities to better inform decision making * SOCIAL - Recognise and meet our corporate responsibilities to the many stakeholders in our business , including shareholders , employees , suppliers , dealers and customers , and neighbouring communities - Understand our contribution to society in the broader context of " sustainable mobility " to support our long term planning . * ENVIRONMENT - Improve environmental performance throughout the business by reducing and preventing pollution , conserving resources , managing materials of concern responsibly , minimising wastes , and implementing effective and efficient processes - Optimise the potential for the use of alternative fuels , renewable energy , and recycled and sustainable materials David M Smith Chief Executive Officer 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Corp/43CorpZara_Environment_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Corp/43CorpZara_Environment_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Corp/43CorpZara_Environment_EN.txt.txt
new file mode 100644
index 0000000..432c691
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Corp/43CorpZara_Environment_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Our Mission Statement Environmental policy Through Zara\u2019s business model , we aim to contribute to the sustainable development of society and that of the environment with which we interacts . The company 's commitment to the environment is included in Inditex Group\u2019s Corporate Responsibility Statement , published on our website : www . inditex . com . The following are some of the objectives and actions included in the framework of the group 's environmental commitment and have a direct impact on our shops and customers : At the store * - We save energy . * - The eco-friendly shop . * - We produce less waste , and recycle . * - Our commitment extends to all our staff . * - An environmentally aware team . We save energy . The eco-friendly store . We are implementing an eco-friendly management model in our shops in order to reduce energy consumption by 20 % , introducing sustainability and efficiency criteria . This management model sets out measures to be applied to all proce
 sses , including the design of the shop itself , the lighting , heating and cooling systems and the possibility of recycling furniture and decoration . We produce less waste and recycle . Recycling hangers and alarms , which are picked up from our shops and processed into other plastic elements , is an example of our waste management policy . Millions of hangers and alarms are processed each year and both the cardboard and plastic used for packaging are also recycled . - Our commitment extends to all our staff . Increased awareness among our team members . We hold In-company awareness campaigns and specific multimedia-based training programmes to educate our staff in sustainable practices , such as limiting energy consumption , using sustainable transport and modifying behaviour patterns . With the product * - We use ecological fabrics . * - Organic cotton . * - We manufacture PVC-free footwear . We use ecological fabrics . Organic cotton . Zara supports organic farming and makes so
 me of its garments out of organic cotton ( 100 % cotton , completely free of pesticides , chemicals and bleach ) . They have specific labels and are easy to spot in our shops . We produce PVC-free footwear . No petroleum derivatives or non-biodegradable materials are used in the production of our footwear ( PVC free ) . In transport - We use biodiesel fuel . Zara 's fleet of lorries , which transport more than 200 million items of clothing a year , use 5 % biodiesel fuel . This allows us to reduce our CO2 emissions by 500 tons . Animal welfare policy All products of animal origin sold in our shops , including fur and leather , come exclusively from animals raised on food farms and under no circumstances come from animals sacrificed exclusively for the sale of their hide . 
\ No newline at end of file


[08/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/170ted_robert_thurman_on_compassion.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/170ted_robert_thurman_on_compassion.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/170ted_robert_thurman_on_compassion.txt
new file mode 100644
index 0000000..def743c
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/170ted_robert_thurman_on_compassion.txt
@@ -0,0 +1,2 @@
+
+I want to open by quoting Einstein 's wonderful statement , just so people will feel at ease that the great scientist of the 20th century also agrees with us , and also calls us to this action . He said , " A human being is a part of the whole , called by us , universe , a part limited in time and space . He experiences himself , his thoughts and feelings , as something separated from the rest , a kind of optical delusion of his consciousness , that separation . This delusion is a kind of prison for us , restricting us to our personal desires and to affection for a few persons nearest to us . Our task must be to free ourselves from this prison by widening our circle of compassion , to embrace all living creatures and the whole of nature in its beauty . " This insight of Einstein 's is uncannily close to that of Buddhist psychology , wherein compassion , karuna , it is called , is defined as , " The sensitivity to another 's suffering and the corresponding will to free the other from
  that suffering . " It pairs closely with love . Which is the will for the other to be happy . Which requires , of course , that one feels some happiness oneself and wishes to share it . This is perfect in that it clearly opposes self-centeredness and selfishness to compassion , the concern for others , and , further , it indicates that those caught in the cycle of self-concern , suffer helplessly , while the compassionate are more free and implicitly more happy . The Dalai Lama often states that compassion is his best friend . It helps him when he is overwhelmed with grief and despair . Compassion helps him turn away from the feeling of his suffering as the most absolute , most terrible suffering anyone has ever had and broadens his awareness of the sufferings of others , even of the perpetrators of his misery and the whole mass of beings . In fact , suffering is so huge and enormous , his own becomes less and less monumental . And he begins to move beyond his self-concern into the
  broader concern for others . And this immediately cheers him up , as his courage is stimulated to rise to the occasion . Thus , he uses his own suffering as a doorway to widening his circle of compassion . He is a very good colleague of Einstein 's , we must say . Now , I want to tell a story , which is a very famous story in the Indian and Buddhist tradition , of the great Saint Asanga who lived -- contemporary of Augustine in the West and was sort of like the Buddhist Augustine . And Asanga lived 800 years after the Buddha 's time . And he was discontented with the state of people 's practice of the Buddhist religion in India at that time . And so he said , " I 'm sick of all this . Nobody 's really living the doctrine . They 're talking about love and compassion and wisdom and enlightenment , but they are acting selfish and pathetic . So Buddha 's teaching has lost its momentum . I know next Buddha will come a few thousand years from now , but exists currently in a certain heave
 n , that 's Maitreya . So , I 'm going to go on a retreat , and I 'm going to meditate and pray until the Buddha Maitreya reveals himself to me , and gives me a teaching or something to revive the practice of compassion in the world today . " So he went on this retreat . And he meditated for three years and he did not see the future Buddha Maitreya . And he left in disgust . And as he was leaving , he saw a man -- a funny little man -- sitting sort of part way down the mountain . And he had a lump of iron . And he was rubbing it with a cloth . And he became became interested in that . He said , " Well what are you doing ? " And the man said , " I 'm making a needle . " And he said , " That 's ridiculous . you ca n't make a needle by rubbing a lump of iron with a cloth . " And the man said , " Really ? " And he showed him a dish full of needles . So he said , " Okay , I get the point . " He went back to his cave . He meditated again . Another three years , no vision . He leaves again
  . This time , he comes down . And as he 's leaving , he sees a bird making a nest on a cliff ledge . And where it 's landing to bring the twigs to the cliff , its feathers brushes the rock , and it had cut the rock in , inches , six to eight inches in , there was a cleft in the rock by the brushing of the feathers of generations of the birds . So he said , " All right . I get the point . " He went back . Another three years . Again , no vision of Maitreya after nine years . And , he again leaves , and this time water dripping , making a giant bowl in the rock where it drips in a stream . And so , again , he goes back . And after 12 there is still no vision . And he 's freaked out . And he wo n't even look left or right to see any encouraging vision . And he comes to the town . He 's a broken person . And there , in the town , he 's approached by a dog who comes like this -- one of these terrible dogs you can see in some poor countries , even in America , I think , in some areas -- 
 and he 's looking just terrible . And he becomes interested in this dog because it 's so pathetic , and it 's trying to attract his attention . And he sits down looking at the dog . And the dog 's whole hindquarters are a complete open sore . And some of it is like gangrenous . And there 's like maggots in the flesh . And it 's terrible . He thinks , " What can I do to fix up this dog ? Well , at least I can clean this wound and wash it . " So he takes it to some water , he 's about to clean , then his awareness focuses on the maggots . And he sees the maggots , and the maggots are kind of looking a little cute . And they 're maggoting happily in the dog 's hindquarters there . " Well , if I clean the dog , I 'll kill the maggots . So how can that be ? That 's it . I 'm a useless person and there 's no Buddha , no Maitreya , and everything is all hopeless . And now I 'm going to kill the maggots ? " So , he had a brilliant idea . And he took a shard of something , and cut a piece of
  flesh from his thigh , and he placed it on ground . He was not really thinking too carefully about the ASPCA . He was just immediately caught with the situation . So he thought , " I will take the maggots and put them on this piece of flesh , then clean the dog 's wounds , and then , you know , I 'll figure out what to do with the maggots . " So he starts to do that . He ca n't grab the maggots . Apparently they wriggle around . They 're kind of hard to grab , these maggots . So he says , " Well , I 'll put my tongue on the dog 's flesh . And then the maggots will jump on my warmer tongue . The dog is kind of used up . And then I 'll spit them one by one down on the thing . " So he goes down , and he 's sticking his tongue out like this . And he had to close his eyes , it 's so disgusting , and the smell and everything . And then , suddenly , there 's a pfft , a noise like that . He jumps back and there , of course , is the future Buddha Maitreya . In a beautiful vision like rainbo
 w lights , golden , jeweled , plasma body , like exquisite mystic vision , he sees . And he says , " Oh . " He bows . But , being human , he 's immediately thinking of his next complaint . So as he comes up from his first bow he says , " My Lord , I 'm so happy to see you , but where have you been for 12 years ? What is this ? " And Maitreya says , " I was with you . Who do you think was making needles and making nests and dripping on rocks for you , mister dense ? " ( Laughter ) " Looking for the Buddha in person . " he said . And he said , " You did n't have , until this moment , real compassion . And , until you have real compassion , you cannot recognize love . " Maitreya means love , the loving one , you know , in Sanskrit . And so he looked very dubious , Asanga did . And he said , " If you do n't believe me , just take me with you . " And so he took the Maitreya -- it shrunk into a globe , a ball -- took him on his shoulder . And he ran into town in the marketplace , and he s
 aid , " Rejoice . Rejoice . The future Buddha has come ahead of all predictions . Here he is . " And then pretty soon they started throwing rocks and stones at him -- It was n't Chautauqua . It was some other town -- because they saw a demented looking , scrawny looking yogi man , like some kind of hippie , with a bleeding leg and a rotten dog on his shoulder , shouting that the future Buddha had come . So , naturally , they chased him out of town . But on the edge of town , one elderly lady , a char woman in the charnel ground , saw a jeweled foot on a jeweled lotus on his shoulder and then the dog , but she saw the jewel foot of the Maitreya , and she offered a flower . So that encouraged him , and he went with Maitreya . With Maitreya then took him to a certain heaven , the way the Buddhist myth unfolds in a typical way . And Maitreya then kept him in heaven for five years , dictating to him five complicated tomes of the methodology of how you cultivate compassion . And then I th
 ought I would share with you what that method is , or one of them . Famous one , it 's called the " Sevenfold Causal Method of Developing Compassion . " And it begins first by one meditating and visualizing that all beings are with one , and all -- even animals too -- but everyone is in human form . The animals are in one of their human lives . The humans are human . And then , among them , you think of your friends and loved ones , the circle at the table . And you think of your enemies , and you think of the neutral ones . And then you try to say , " Well , the loved ones I love . But , you know , after all , they 're nice to me . I had fights with them . Sometimes they were unfriendly . I got mad . Brothers can fight . Parents and children can fight . So , in a way , I like them so much because they 're nice to me . While the neutral ones I do n't know . They could all be just fine . And then the enemies I do n't like because they 're mean to me . But they are nice to somebody . 
 I could be them . " And then the Buddhists , of course , think , because we 've all had infinite previous lives , the Buddhists think that we 've all been each other 's relatives , actually , and everyone , therefore all of you , in the Buddhist view in some previous life , although you do n't remember it and neither do I , have been my mother , for which I do apologize for the trouble I caused you . And also , actually , I 've been your mother . I 've been female , and I 've been every single one of you , your mother in a previous life , the way the Buddhists reflect . So , my mother is this life is really great . But all of you in a way are part of the eternal mother . You gave me that expression , the eternal mama , you said . That 's wonderful . So , that 's the way the Buddhists do it . A theist , Christian , can think that all beings , even my enemies , are God 's children . So , in that sense , we 're related . So , they first create this foundation of equality . So , we sort
  of reduce a little of the clinging to the ones we love -- just in the meditation -- and we open our mind to those we do n't know . And we definitely reduce the hostility and " I do n't want to be compassionate to them " to the ones we think of as the bad guys , the ones we hate and we do n't like . And we do n't hate anyone therefore . So we equalize . That 's very important . And then the next thing we do is what is called mother recognition . And that is , we think of every being as familiar , as family . We expand . We take the feeling about remembering a mama , and we defuse that to all beings in this meditation . And we see the mother in every being . We see that look that the mother has on her face , this looking at this child that is a miracle that she has produced from her own body , being a mammal , where she has true compassion , truly is the other , and identifies completely . Often the life of that other will be more important to her than her own life . And that 's why 
 it 's the most powerful form of -- altruism . The mother is what is the model of all altruism for human beings , in spiritual traditions . And so , we reflect until we can sort of see that motherly expression in all beings . People laugh at me because , you know , I used to say that I used to meditate on mama Cheney as my mom , when , of course , I was annoyed with him about all of his evil doings in Iraq . I used to meditate on George Bush . He 's quite a cute mom in a female form . Has his little ears and he smiles and he rocks you in his arms . And you think of him as nursing you . And then Saddam Hussein 's serious mustache is a problem . But you think of him as a mom . And this is the way you do it . You take any being who looks weird to you , and you see how they could be familiar to you . And you do that for awhile until you really feel that . You can feel the familiarity of all beings . Nobody seems alien . They 're not " other . " You reduce the feeling of otherness about b
 eings . Then you move from there to remembering the kindness of mothers in general , if you can remember the kindness of your own mother , if you can remember the kindness of your spouse , or , if you are a mother yourself , how you were with your children . And you begin to get very sentimental , you cultivate sentimentality intensely . You will even weep , perhaps , with gratitude and kindness . And then you connect that with your feeling that everyone has that motherly possibility . Every being , even the most mean looking ones , can be motherly . And then , third , you step from there to what is called a feeling of gratitude . You want to repay that kindness that all beings have shown to you . And then the fourth step , you go to what is called lovely love . In each one of these you can take some weeks , or months , or days depending on how you do it , or you can do them in a run , this meditation . And then you think of how lovely beings are when they are happy , when they are 
 satisfied . And every being looks beautiful when they are internally feeling a happiness . Their face does n't look like this . When they 're angry , they look ugly , every being , but when they 're happy they look beautiful . And so you see beings in their potential happiness . And you feel a love toward them that you want them to be happy , even the enemy . And , actually , it 's very logical to want to -- we think Jesus is being unrealistic when he says love thine enemy . He does say that , and we think he 's being unrealistic and sort of spiritual and highfalutin and , " Nice for him to say it , but I ca n't do that . " But , actually , that 's practical . If you love your enemy that means you want your enemy to be happy . If your enemy was really happy , why would they bother to be your enemy ? How boring to run around chasing you . They would be relaxing somewhere having a good time . So it makes sense to want your enemy to be happy because they 'll stop being your enemy becau
 se that 's too much trouble . But anyway , that 's the lovely love . And then finally , the fifth step is compassion , universal compassion . And that is where you then look at the reality of all the beings you can think of . And you look at them , and you see how they are . And you realize how unhappy they are actually , mostly , most of the time . You see that furrowed brow in people . And then you realize they do n't even have compassion on themselves . They 're driven by this duty and this obligation . " I have to get that . I need more . I 'm not worthy . And I should do something . " And they 're rushing around all stressed out . And they think of it as somehow macho , hard discipline on themselves . But actually they are cruel to themselves . And , of course , they are cruel and ruthless toward others . And they , then , never get any positive feedback . And the more they succeed , and the more power they have , the more unhappy they are . And this is where you feel real comp
 assion for them . And you then feel you must act . And it 's the motivation -- And the choice of action , of course , hopefully will be more practical than poor Asanga who was fixing the maggots on the dog , because he had that motivation , and whoever was in front of him , he wanted to help . But , of course , that is impractical . He should have founded the ASPCA in the town and gotten some scientific help for dogs and maggots . And I 'm sure he did that later . But that just indicates the state of mind , you know . And so the next step -- the sixth step beyond universal compassion -- which then is this thing where you 're linked with the needs of others in a true way , and you have compassion for yourself also , and you do n't -- it is n't sentimental only . You might be in fear of something . Some bad guy is making himself more and more unhappy being more and more mean to other people and getting punished in the future for it in various ways . And in Buddhism , they catch it in 
 the future life . Of course in theistic religion they 're punished by God or whatever . And materialism , they think they get out of it just by not existing , by dying , but they do n't . And so they get reborn as whatever , you know . Never mind . I wo n't get into that . But the next step is called universal responsibility . And that is very important -- the Charter of Compassion must lead us to develop through true compassion , what is called universal responsibility . And that means that the great teaching of his holiness , the Dalai Lama , that he always teaches everywhere , and he says that is the common religion of humanity , kindness , But kindness means universal responsibility . And that means whatever happens to other beings is happening to us , that we are responsible for that , and we should take it and do whatever we can at whatever little level and small level that we can do it . We absolutely must do that . There is no way not to do it . And then , finally , that lea
 ds to a new orientation in life where we live equally for ourselves and others , and we realize that happiness for ourselves -- and we are joyful and happy . One thing we must n't think is compassion makes you miserable . Compassion makes you happy . The first person who is happy , when you get great compassion , is yourself , even if you have n't done anything yet for anybody else . Although , the change in your mind already does something for other beings . They can sense this new quality in yourself , and it helps them already , and gives them an example . And that uncompassionate clock has just showed me that it 's all over . So , practice compassion , read the charter , disseminate it and develop it within yourself . Do n't just think , oh well , I 'm compassionate , or I 'm not compassionate , and sort of think you 're stuck there . You can develop this . You can diminish the non-compassion , the cruelty , the callousness , the neglect of others . Take universal responsibility
  for them , and then , not only will God smile and the eternal mama will smile , but Karen Armstrong will smile . Thank you very much . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/171ted_rory_sutherland_life_lessons_from_an_ad_man.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/171ted_rory_sutherland_life_lessons_from_an_ad_man.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/171ted_rory_sutherland_life_lessons_from_an_ad_man.txt
new file mode 100644
index 0000000..6d0bcf8
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/171ted_rory_sutherland_life_lessons_from_an_ad_man.txt
@@ -0,0 +1,2 @@
+
+This is my first time at TED . Normally , as an advertising man , I actually speak at TED Evil , which is TED 's secret sister organization -- the one that pays all the bills . It 's held every two years in Burma . And I particularly remember a really good speech by Kim Jong Il on how to get teens smoking again . ( Laughter ) But , actually , it 's suddenly come to me after years working in the business , that what we create in advertising , which is intangible value -- you might call it perceived value , you might call it badge value , subjective value , intangible value of some kind -- gets rather a bad rap . If you think about it , if you want to live in a world in the future where there are fewer material goods , you basically have two choices . You can either live in a world which is poorer , which people in general do n't like . Or you can live in a world where actually intangible value constitutes a greater part of overall value , that actually intangible value , in many ways
  is a very , very fine substitute for using up labor or limited resources in the creation of things . Here is one example . This is a train which goes from London to Paris . The question was given to a bunch of engineers , about 15 years ago , " How do we make the journey to Paris better ? " And they came up with a very good engineering solution , which was to spend six billion pounds building completely new tracks from London to the coast , and knocking about 40 minutes off a three-and-half-hour journey time . Now , call me Mister Picky . I 'm just an ad man ... ... but it strikes me as a slightly unimaginative way of improving a train journey merely to make it shorter . Now what is the hedonic opportunity cost on spending six billion pounds on those railway tracks ? Here is my naive advertising man 's suggestion . What you should in fact do is employ all of the world 's top male and female supermodels , pay them to walk the length of the train , handing out free Chateau Petrus for
  the entire duration of the journey . ( Laughter ) ( Applause ) Now , you 'll still have about three billion pounds left in change , and people will ask for the trains to be slowed down . ( Laughter ) Now , here is another naive advertising man 's question again . And this shows that engineers , medical people , scientific people , have an obsession with solving the problems of reality , when actually most problems , once you reach a basic level of wealth in society , most problems are actually problems of perception . So I 'll ask you another question . What on earth is wrong with placebos ? The seem fantastic to me . They cost very little to develop . They work extraordinarily well . They have no side effects , or if they do , they 're imaginary , so you can safely ignore them . ( Laughter ) So I was discussing this . And I actually went to the Marginal Revolution blog by Tyler Cowen . I do n't know if anybody knows it . Someone was actually suggesting that you can take this conce
 pt further , and actually produce placebo education . The point is that education does n't actually work by teaching you things . It actually works by giving you the impression that you 've had a very good education , which gives you an insane sense of unwarranted self confidence , which then makes you very , very successful in later life . So , welcome to Oxford , ladies and gentlemen . ( Laughter ) ( Applause ) But , actually , the point of placebo education is interesting . How many problems of life can be solved actually by tinkering with perception , rather than that tedious , hardworking and messy business of actually trying to change reality ? Here 's a great example from history . I 've heard this attributed to several other kings , but doing a bit of historical research it seems to be Fredrick the Great . Fredrick the Great of Prussia was very very keen for the Germans to adopt the potato , and to eat it . Because he realized that if you had two sources of carbohydrate , wh
 eat and potatoes , you get less price volatility in bread . And you get a far lower risk of famine , because you actually had two crops to fall back on , not one . The only problem is : potatoes , if you think about it , look pretty disgusting . And also , 18th century Prussians ate very , very few vegetables -- rather like contemporary Scottish people . ( Laughter ) So , actually , he tried making it compulsory . The Prussian peasantry said , " We ca n't even get the dogs to eat these damn things . They are absolutely disgusting and they 're good for nothing . " There are even records of people being executed for refusing to grow potatoes . So he tried plan B. He tried the marketing solution , which is he declared the potato as a royal vegetable . And none but the royal family could consume it . And he planted it in a royal potato patch , with guards who had instructions to guard over it , night and day , but with secret instructions not to guard it very well . ( Laughter ) Now 18t
 h century peasants know that there is one pretty safe rule in life , which is if something is worth guarding , it 's worth stealing . Before long , there was a massive underground potato-growing operation in Germany . What he 'd effectively done is he 'd re-branded the potato . It was an absolute masterpiece . I told this story and a gentleman from Turkey came up to me and said , " Very , very good marketer , Fredrick the Great . But not a patch on Ataturk . " Ataturk , rather like Nicolas Sarkozy , was very keen to discourage the wearing of a veil , in Turkey , to modernize it . Now , boring people would have just simply banned the veil . But that would have ended up with a lot of awful kickback and a hell of a lot of resistance . Ataturk was a lateral thinker . He made it compulsory for prostitutes to wear the veil . ( Laughter ) ( Applause ) I ca n't verify that fully . But it does not matter . There is your environmental problem solved , by the way , guys : All convicted child m
 olesters have to drive a Porsche Cayenne . ( Laughter ) What Ataturk realized actually is two very fundamental things . Which is that , actually , first one , all value is actually relative . All value is perceived value . For those of you who do n't speak Spanish , jugo de naranja -- it 's actually the Spanish for " orange juice . " Because actually it 's not the dollar . It 's actually the peso in Buenos Aires . Very clever Buenos Aires street vendors decided to practice price discrimination to the detriment to any passing gringo tourists . As an advertising man , I have to admire that . But the first thing this all shows is that all value is subjective . Second point is that persuasion is often better than compulsion . These funny signs that flash your speed at you , some of the new ones , on the bottom right , now actually show a smiley face or a frowny face , to act as an emotional trigger . What 's fascinating about these signs is they cost about 10 percent of the running cost
  of a conventional speed camera . But they prevent twice as many accidents . So , the bizarre thing which is baffling to conventional , classically trained economists , is that a weird little smiley face has a better effect on changing your behavior than the threat of a �60 fine and three penalty points . Tiny little behavioral economics detail : in Italy , penalty points go backwards . You start with 12 and they take them away . Because the found that loss aversion is a more powerful influence on people 's behavior . In Britain we tend to feel , " Whoa ! Got another three ! " Not so in Italy . Another fantastic case of creating intangible value to replace actual or material value , which remember , is what , after all , the environmental movement needs to be about : This , again , is from Prussia , from , I think , about 1812 , 1813. The wealthy Prussians , to help in war against the French , were encouraged to give in all their jewelry . And it was replaced with replica jewelry m
 ade of cast iron . Here 's one : " Gold gab ich f�r Eisen , 1813. " The interesting thing is that for 50 years hence , the highest status jewelry you could wear in Prussia was n't made of gold or diamonds . It was made of cast iron . Because actually , never mind the actual intrinsic value of having gold jewelry . This actually had symbolic value , badge value . It said that your family had made a great sacrifice in the past . So , the modern equivalent would of course be this . ( Laughter ) But , actually , there is a thing , just as there are Veblen goods , where the value of the good depends on it being expensive and rare -- there are opposite kind of things where actually the value in them depends on them being ubiquitous , classless and minimalistic . If you think about it , Shakerism was a proto-environmental movement . Adam Smith talks about 18th century America where the prohibition against visible displays of wealth was so great , it was almost a block in the economy in Ne
 w England , because even wealthy farmers could find nothing to spend their money on , without incurring the displeasure of their neighbors . It 's perfectly possible to create these social pressures which lead to more egalitarian societies . What 's also interesting , if you look at products that have a high component of what you might call messaging value , a high component of intangible value , versus their intrinsic value : They are often quite egalitarian . In terms of dress , denim is perhaps the perfect example of something which replaces material value with symbolic value . Coca-Cola . A bunch of you may be a load of pinkos , and you may not like the Coca-Cola company . But it 's worth remembering Andy Warhol 's point about Coke . What Warhol said about Coke is , he said , " What I really like about Coca-Cola is the president of the United States ca n't get a better Coke than the bum on the corner of the street . " Now , that is , actually , when you think about it , we take 
 it for granted -- it 's actually a remarkable achievement , to produce something that 's so democratic . Now , we basically have to change our views slightly . There is a basic view that real value involves making things , involves labor . It involves engineering . It involves limited raw materials . And that what we add on top is kind of false . It 's a fake version . And there is a reason for some suspicion and uncertainly about it . It patently veers toward propaganda . However , what we do have now is a much more variegated media ecosystem in which to kind of create this kind of value . And it 's much fairer . When I grew up , this was basically the media environment of my childhood as translated into food . You had a monopoly supplier . On the left , you have Rupert Murdoch , or the BBC . ( Laughter ) And on your right you have a dependent public which is pathetically grateful for anything you give it . ( Laughter ) Nowadays , the user is actually involved . This is actually wh
 at 's called , in the digital world , " user-generated content . " Although it 's called agriculture , in the world of food . ( Laughter ) This is actually called a mash-up , where you take content that someone else has produced and you do something new with it . In the world of food we call it cooking . This is food 2.0 , which is food you produce for the purpose of sharing it with other people . This is mobile food . British are very good at that . Fish and chips in newspaper , the Cornish Pastie , the pie , the sandwich . We invented the whole lot of them . We 're not very good at food in general . Italians do great food , but it 's not very portable , generally . ( Laughter ) I only learned this the other day . The Earl of Sandwich did n't invent the sandwich . He actually invented the toasty . But then , the Earl of Toasty would be a ridiculous name . ( Laughter ) Finally , we have contextual communication . Now , the reason I show you Pernod -- it 's only one example . Every c
 ountry has a contextual alcoholic drink . In France it 's Pernod . It tastes great within the borders of that country . But absolute shite if you take it anywhere else . ( Laughter ) Unicum in Hungary , for example . The Greeks have actually managed to produce something called Retsina , which even tastes shite when you 're in Greece . ( Laughter ) But so much communication now is contextual that the capacity for actually nudging people , for giving them better information -- B. J. Fogg , at the University of Stanford , makes the point that actually the mobile phone is -- He 's invented the phrase , " persuasive technologies . " He believes the mobile phone , by being location-specific , contextual , timely and immediate , is simply the greatest persuasive technology device ever invented . Now , if we have all these tools at our disposal , we simply have to ask the question , and Thaler and Sunstein have , of how we can use these more intelligently . I 'll give you one example . If y
 ou had a large red button of this kind , on the wall of your home , and every time you pressed it it saved 50 dollars for you , put 50 dollars into your pension , you would save a lot more . The reason is that the interface fundamentally determines the behavior . Okay ? Now , marketing has done a very very good job of creating opportunities for impulse buying . Yet we 've never created the opportunity for impulse saving . If you did this , more people would save more . It 's simply a question of changing the interface by which people make decisions . And the very nature of the decisions changes . Obviously , I do n't want people to do this , because as an advertising man I tend to regard saving as just consumerism needlessly postponed . ( Laughter ) But if anybody did want to do that , that 's the kind of thing we need to be thinking about , actually : fundamental opportunities to change human behavior . Now , I 've got an example here from Canada . There was a young intern at Ogilv
 y Canada called Hunter Somerville , who was working in improv in Toronto , and got a part-time job in advertising , and was given the job of advertising Shreddies . Now this is the most perfect case of creating intangible added value , without changing the product in the slightest . Shreddies is a strange , square , whole-grain cereal , only available in New Zealand , Canada and Britain . It 's Kraft 's peculiar way of rewarding loyalty to the crown . ( Laughter ) In working out how you could relaunch Shreddies , he came up with this . Video : ( Buzzer ) Man : Shreddies is supposed to be square . ( Laughter ) Woman : Have any of these diamond shapes gone out ? ( Laughter ) Voiceover : New Diamond Shreddies cereal . Same 100 percent whole-grain wheat in a delicious diamond shape . ( Applause ) Rory Sutherland : I 'm not sure this is n't the most perfect example of intangible value creation . All it requires is photons , neurons , and a great idea to create this thing . I would say it
  's a work of genius . But , naturally , you ca n't do this kind of thing without a little bit of market research . Man : So , Shreddies is actually producing a new product , which is something very exciting for them . So they are introducing new Diamond Shreddies . ( Laughter ) So I just want to get your first impressions when you see that , when you see the Diamond Shreddies box there . ( Laughter ) Woman : Were n't they square ? Woman #2 : I 'm a little bit confused . Woman #3 : They look like the squares to me . Man : They -- Yeah , it 's all in the appearance . But it 's kind of like flipping a six or a nine like a six . If you flip it over it looks like a nine . But a six is very different from a nine . Woman # 3 : Or an " M " and a " W " . Man : An " M " and a " W " , exactly . Man #2 : [ unclear ] You just looked like you turned it on its end . But when you see it like that it 's more interesting looking . Man : Just try both of them . Take a square one there , first . ( Lau
 ghter ) Man : Which one did you prefer ? Man #2 : The first one . Man : The first one ? ( Laughter ) Rory Sutherland : Now , naturally , a debate raged . There were conservative elements in Canada , unsurprisingly , who actually resented this intrusion . So , eventually , the manufacturers actually arrived at a compromise , which was the combo pack . ( Laughter ) ( Applause ) ( Laughter ) If you think it 's funny , bear in mind there is an organization called the American Institute of Wine Economics , which actually does extensive research into perception of things , and discovers that except for among perhaps five or ten percent of the most knowledgeable people , there is no correlation between quality and enjoyment in wine , except when you tell the people how expensive it is , in which case they tend to enjoy the more expensive stuff more . So drink your wine blind in the future . But this is both hysterically funny -- but I think an important philosophical point , which is , goi
 ng forward , we need more of this kind of value . We need to spend more time appreciating what already exists , and less time agonizing over what else we can do . Two quotations to more or less end with . One of them is , " Poetry is when you make new things familiar and familiar things new . " Which is n't a bad definition of what our job is , to help people appreciate what is unfamiliar , but also to gain a greater appreciation , and place a far higher value on those things which are already existing . There is some evidence , by the way , that things like social networking help do that . Because they help people share news . They give badge value to everyday little trivial activities . So they actually reduce the need for actually spending great money on display , and increase the kind of third-party enjoyment you can get from the smallest , simplest things in life . Which is magic . The second one is the second G. K. Chesterton quote of this session , which is , " We are perishi
 ng for want of wonder , not for want of wonders , " which I think for anybody involved in technology , is perfectly true . And a final thing : When you place a value on things like health , love , sex and other things , and learn to place a material value on what you 've previously discounted for being merely intangible , a thing not seen , you realize you 're much much wealthier than you ever imagined . Thank you very much indeed . ( Applause ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/172ted_sean_gourley_on_the_mathematics_of_war.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/172ted_sean_gourley_on_the_mathematics_of_war.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/172ted_sean_gourley_on_the_mathematics_of_war.txt
new file mode 100644
index 0000000..7d554ee
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/172ted_sean_gourley_on_the_mathematics_of_war.txt
@@ -0,0 +1,2 @@
+
+We look around the media , as we see on the news from Iraq , Afghanistan , Sierra Leone , and the conflict seems incomprehensible to us . And that 's certainly how it seemed to me when I started this project . But as a physicist , I thought , well if you give me some data , I could maybe understand this . You know , give us a go . So as a naive New Zealander I thought , well I 'll go to the Pentagon . Can you get me some information ? ( Laughter ) No. So I had to think a little harder . And I was watching the news one night in Oxford . And I looked down at the chattering heads on my channel of choice . And I saw that there was information there . There was data within the streams of news that we consume . All this noise around us actually has information . So what I started thinking was , perhaps there is something like open source intelligence here . If we can get enough of these streams of information together we can perhaps start to understand the war . So this is exactly what I 
 did . We started bringing a team together , an interdisciplinary team of scientists , of economists , mathematicians . We brought these guys together and we started to try and solve this . We did it in three steps . The first step we did was to collect . We did 130 different sources of information -- from NGO reports to newspapers and cable news . We brought this raw data in and we filtered it . We extracted the key bits on information to build the database . That database contained the timing of attacks , the location , the size and the weapons used . It 's all in the streams of information we consume daily , we just have to know how to pull it out . And once we had this we could start doing some cool stuff . What if we were to look at the distribution of the sizes of attacks ? What would that tell us ? So we started doing this . And you can see here on the horizontal axis you 've got the number of people killed in an attack or the size of the attack . And on the vertical axis you 
 've got the number of attacks . So we plot data for sample on this . You see some sort of random distribution -- perhaps 67 attacks , one person was killed , or 47 attacks where seven people were killed . We did this exact same thing for Iraq . And we did n't know , for Iraq what we were going to find . It turns out what we found was pretty surprising . You take all of the conflict , all of the chaos , all of the noise , and out of that comes this precise mathematical distribution of the way attacks are ordered in this conflict . This blew our mind . Why should a conflict like Iraq have this as its fundamental signature ? Why should there be order in war ? We did n't really understand that . We thought maybe there is something special about Iraq . So we looked at a few more conflicts . We looked at Colombia , we looked at Afghanistan , and we looked at Senegal . And the same pattern emerged in each conflict . This was n't supposed to happen . These are different wars , with differen
 t religious factions , different political factions , and different socioeconomic problems . And yet the fundamental patterns underlying them are the same . So we went a little wider . We looked around the world at all the data we could get our hands on . From Peru to Indonesia , we studied this same pattern again . And we found that not only were the distributions these straight lines , but the slope of these lines , they clustered around this value of Alpha equals 2.5 . And we could generate an equation that could predict the likelihood of an attack . What we 're saying here is the probability of an attack killing X number of people in a country like Iraq , is equal to a constant , times the size of that attack , raised to the power of negative Alpha . And negative Alpha is the slope of that line I showed you before . So what ? This is data , statistics . What does it tell us about these conflicts ? That was a challenge we had to face as physicists . How do we explain this ? And w
 hat we really found was that Alpha if we really think about it , is the organizational structure of the insurgency . Alpha is the distribution of the sizes of attacks , which is really the distribution of the group strength carrying out the attacks . So we look at a process of group dynamics -- coalescence and fragmentation . Groups coming together . Groups breaking apart . And we start running the numbers on this . Can we simulate it ? Can we create the kind of patterns that we 're seeing in places like Iraq ? Turns out we kind of do a reasonable job . We can run these simulations . We can recreate this using a process of group dynamics to explain the patterns that we see all around the conflicts around the world . So what 's going on ? Why should these different -- seemingly different conflicts have the same patterns ? Now what I believe is going on is that the insurgent forces , they evolve over time . They adapt . And it turns out there is only one solution to fight a much stron
 ger enemy . And if you do n't find that solution as an insurgent force , you do n't exist . So every insurgent force that is ongoing , every conflict that is ongoing , it 's going to look something like this . And that is what we think is happening . Taking it forward , how do we change it ? How do we end a war like Iraq ? What does it look like ? Alpha is the structure . It 's got a stable state at 2.5 . This is what wars look like when they continue . We 've got to change that . We can push it up . The forces become more fragmented . There is more of them , but they are weaker . Or we push it down . They 're more robust . There is less groups . But perhaps you can sit and talk to them . So this graph here , I 'm going to show you now . No one has seen this before . This is literally stuff that we 've come through last week . And we see the evolution of Alpha through time . We see it start . And we see it grow up to the stable state the wars around the world look like . And it stay
 s there through the invasion of Falusia until the Samarra bombings in the Iraqi elections of '06 . And the system gets perturbed . It moves upwards to a fragmented state . This is when the surge happens . And depending on who you ask , the surge was supposed to push it up even further . The opposite happened . The groups became stronger . They became more robust . And so I 'm thinking , right , great , it 's going to keep going down . We can talk to them . We can get a solution . The opposite happened . It 's moved up again . The groups are more fragmented . And this tells me one of two things . Either we 're back where we started , and the surge has had no effect . Or finally the groups have been fragmented to the extent that we can start to think about maybe moving out . I do n't know what the answer is to that . But I know that we should be looking at the structure of the insurgency to answer that question . Thank you . ( Applause ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/173ted_siegfried_woldhek_shows_how_he_found_the_true_face_of_leonardo.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/173ted_siegfried_woldhek_shows_how_he_found_the_true_face_of_leonardo.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/173ted_siegfried_woldhek_shows_how_he_found_the_true_face_of_leonardo.txt
new file mode 100644
index 0000000..5871249
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/173ted_siegfried_woldhek_shows_how_he_found_the_true_face_of_leonardo.txt
@@ -0,0 +1,2 @@
+
+Good morning . Let 's look for a minute at the greatest icon of all , Leonardo da Vinci . We 're all familiar with his fantastic work -- his drawings , his paintings , his inventions , his writings . But we do not know his face . Thousands of books have been written about him , but there 's controversy , and it remains , about his looks . Even this well-known portrait is not accepted by many art historians . So what do you think ? Is this the face of Leonardo da Vinci or is n't it ? Let 's find out . Leonardo was a man that drew everything around him . He drew people , anatomy , plants , animals , landscapes , buildings , water , everything . But no faces ? I find that hard to believe . His contemporaries made faces , like the ones you see here . En face or three quarters . So surely a passionate drawer like Leonardo must have made self-portraits from time to time . So let 's try to find them . I think that if we were to scan all of his work and look for self-portraits , we would fi
 nd his face looking at us . So I looked at all of his drawings , more than 700 , and looked for male portraits . There are about 120 , you see them here . Which ones of these could be self-portraits ? Well , for that they have to be done as we just saw , en face or three-quarters . So we can eliminate all the profiles . It also has to be sufficiently detailed . So we can also eliminate the ones that are very vague or very stylized . And we know from his contemporaries that Leonardo was a very handsome , even beautiful man . So we can also eliminate the ugly ones or the caricatures . ( Laughter ) And look what happens -- only three candidates remain that fit the bill . And here they are . Yes indeed , the old man is there , as is this famous pen drawing of the Homo Vitruvianos . And lastly , the only portrait of a male that Leonardo painted , " The Musician . " Before we go into these faces , I should explain why I have some right to talk about them . I 've made more than 1,100 portr
 aits myself for newspapers , over the course of 300 -- 30 years , sorry , 30 years only . ( Laughter ) But there are 1,100 , and very few artists have drawn so many faces . So I know a little about drawing and analyzing faces . OK , now let 's look at these three portraits . And hold onto your seats , because if we zoom in on those faces , remark how they have the same broad forehead , the horizontal eyebrows , the long nose , the curved lips and the small , well-developed chin . I could n't believe my eyes when I first saw that . There is no reason why these portraits should look alike . All we did was look for portraits that had the characteristics of a self-portrait , and look , they are very similar . Now , are they made in the right order ? The young man should be made first . And as you see here from the years that they were created , it is indeed the case . They are made in the right order . What was the age of Leonardo at the time ? Does that fit ? Yes it does . He was 33 , 
 38 and 63 when these were made . So we have three pictures , potentially of the same person of the same age as Leonardo at the time . But how do we know it 's him , and not someone else ? Well , we need a reference . And here 's the only picture of Leonardo that 's widely accepted . It 's a statue made by Verrocchio , of David , for which Leonardo posed as a boy of 15. And if we now compare the face of the statue , with the face of the musician , you see the very same features again . The statue is the reference , and it connects the identity of Leonardo to those three faces . Ladies and gentlemen , this story has not yet been published . It 's only proper that you here at TED hear and see it first . The icon of icons finally has a face . Here he is -- Leonardo da Vinci . ( Applause ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/174ted_stephen_wolfram_computing_a_theory_of_everything.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/174ted_stephen_wolfram_computing_a_theory_of_everything.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/174ted_stephen_wolfram_computing_a_theory_of_everything.txt
new file mode 100644
index 0000000..97cf3ef
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/174ted_stephen_wolfram_computing_a_theory_of_everything.txt
@@ -0,0 +1,2 @@
+
+So I want to talk today about an idea . It 's a big idea . Actually , I think it 'll eventually be seen as probably the single biggest idea that 's emerged in the past century . It 's the idea of computation . Now , of course , that idea has brought us all of the computer technology we have today and so on . But there 's actually a lot more to computation than that . It 's really a very deep , very powerful , very fundamental idea , whose effects we 've only just begun to see . Well , I myself have spent the past 30 years of my life working on three large projects that really try to take the idea of computation seriously . So I started off at a young age as a physicist using computers as tools . Then , I started sort of drilling down , thinking about the computations I might want to do , trying to figure out what primitives they could be built up from and how they could be automated as much as possible . Eventually , I created a whole structure based on symbolic programming and so o
 n that let me build Mathematica . And for the past 23 years , at an increasing rate , we 've been pouring more and more ideas and capabilities and so on into Mathematica , and I 'm happy to say that 's led to many good things in R and D and education , lots of other areas . Well , I have to admit , actually , that I also had a very selfish reason for building Mathematica . I wanted to use it myself , a bit like Galileo got to use his telescope 400 years ago . But I wanted to look , not at the astronomical universe , but at the computational universe . So we normally think of programs as being complicated things that we build for very specific purposes . But what about the space of all possible programs ? Here 's a representation of a really simple program . So , if we run this program , this is what we get . Very simple . So let 's try changing the rule for this program a little bit . Now we get another result , still very simple . Try changing it again . You get something a little 
 bit more complicated , but if we keep running this for awhile , we find out that , although the pattern we get is very intricate , it has a very regular structure . So the question is : Can anything else happen ? Well , we can do a little experiment . Let 's just do a little mathematical experiment , try and find out . Let 's just run all possible programs of the particular type that we 're looking at . They 're called cellular automata . You can see a lot of diversity in the behavior here . Most of them do very simple things . But if you look along all these different pictures , at rule number 30 , you start to see something interesting going on . So let 's take a closer look at rule number 30 here . So here it is . We 're just following this very simple rule at the bottom here , but we 're getting all this amazing stuff . It 's not at all what we 're used to , and I must say that , when I first saw this , it came as a huge shock to my intuition , and , in fact , to understand it ,
  I eventually had to create a whole new kind of science . ( Laughter ) This science is different , more general , than the mathematics-based science that we 've had for the past 300 or so years . You know , it 's always seemed like a big mystery how nature , seemingly so effortlessly manages to produce so much that seems to us so complex . Well , I think we 've found its secret . It 's just sampling what 's out there in the computational universe and quite often getting things like Rule 30 or like this . And knowing that , starts to explain a lot of long-standing mysteries in science . It also brings up new issues though , like computational irreducibility . I mean , we 're used to having science let us predict things , but something like this is fundamentally irreducible . The only way to find its outcome is , effectively , just to watch it evolve . It 's connected to , what I call , the principle of computational equivalence , which tells us that even incredibly simple systems can
  do computations as sophisticated as anything . It does n't take lots of technology or biological evolution to be able to do arbitrary computation , just something that happens , naturally , all over the place . Things with rules as simple as these can do it . Well , this has deep implications about the limits of science , about predictability and controllability of things like biological processes or economies , about intelligence in the universe , about questions like free will and about creating technology . You know , working on this science for many years , I kept wondering , " What will be its first killer app ? " Well , ever since I was a kid , I 'd been thinking about systematizing knowledge and somehow making it computable . People like Leibniz had wondered about that too 300 years earlier . But I 'd always assumed that to make progress , I 'd essentially have to replicate a whole brain . Well , now I got to thinking : This scientific paradigm of mine suggests something dif
 ferent . And , by the way , I 've now got huge computation capabilities in Mathematica , and I 'm a CEO with some worldly resources to do large , seemingly crazy , projects . So I decided to just try to see how much of the systematic knowledge that 's out there in the world we can make computable . So , it 's been a big , very complex project , which I was not sure was going to work at all . But I 'm happy to say that it 's actually going really well . And last year we were able to release the first website version of Wolfram Alpha . It 's purpose is to be a serious knowledge engine that computes answers to questions . So let 's give it a try . Let 's start off with something really easy . Hope for the best . Very good . Okay . So far so good . ( Laughter ) Let 's try something a little bit harder . Let 's say ... Let 's do some mathy thing and with luck it 'll work out the answer and try and tell us some interesting things things about related math . We could ask it something about
  the real world . Let 's say -- I do n't know -- What 's the GDP of Spain ? And it should be able to tell us that . Now we could compute something related to this , let 's say the GDP of Spain divided by , I do n't know , the -- hmmm ... let 's say the revenue of Microsoft . ( Laughter ) The idea is that we can sort of just type this in , this kind of question in however we think of it . So let 's try asking a question , like a health related question . So let 's say we have a lab finding that -- you know , we have an LDL level of 140 for a male aged 50. So let 's type that in , and now Wolfram Alpha will go and use available public health data and try to figure out what part of the population that corresponds to and so on . Or let 's try asking about , I do n't know , the international space station . And what 's happening here is that Wolfram Alpha is not just looking up something ; it 's computing , in real time , where the international space station is right now , at this momen
 t , how fast it 's going and so on . So Wolfram Alpha knows about lots and lots of kinds of things . It 's got by now , pretty good coverage of everything you might find in a standard reference library and so on . But the goal is to go much further and , very broadly , to democratize all of this kind of knowledge , and to try and be an authoritative source in all areas , to be able to compute answers to specific questions that people have , not by searching what other people may have written down before , but by using built in knowledge to compute fresh new answers to specific question . Now , of course , Wolfram Alpha is a monumentally huge , long term project with lots and lots of challenges . For a start , one has to curate a zillion different sources of facts and data , and we built quite a pipeline of Mathematica automation and human domain experts for doing this . But that 's just the beginning . Given raw facts or data to actually answer questions , one has to compute , one h
 as to implement all those methods and models and algorithms and so on that science and other areas have built up over the centuries . Well , even starting from Mathematica , this is still a huge amount of work . So far , there are about 8 million lines of Mathematica code in Wolfram Alpha built by experts from many , many different fields . Well , a crucial idea of Wolfram Alpha is that you can just ask it questions using ordinary human language , which means that we 've got to be able to take all those strange utterances that people type into the input field and understand them . And I must say that I thought that step might just be plain impossible . Two big things happened . First , a bunch of new ideas about linguistics that came from studying the computational universe . And second , the realization that having actual computable knowledge completely changes how one can set about understanding language . And , of course , now with Wolfram Alpha actually out in the wild , we can 
 learn from its actual usage . And , in fact , there 's been an interesting coevolution that 's been going on between Wolfram Alpha and its human users . And it 's really encouraging . Right now , if we look at web queries , more than 80 percent of them get handled successfully the first time . And if you look at things like the iPhone app , the fraction is considerably larger . So , I 'm pretty pleased with it all . But , in many ways , we 're still at the very beginning with Wolfram Alpha . I mean , everything is scaling up very nicely . We 're getting more confident . You can expect to see Wolfram Alpha technology showing up in more and more places , working both with this kind of public data , like on the website , and with private knowledge for people and companies and so on . You know , I 've realized that Wolfram Alpha actually gives one a sort of whole new kind of computing that one can call knowledge-based computing , in which one 's starting , not just from raw computation 
 , but from a vast amount of built-in knowledge . And when one does that , one really changes the economics of delivering computational things , whether it 's on the web or elsewhere . You know , we have a fairly interesting situation right now . On the one hand , we have Mathematica , with its sort of precise , formal language and a huge network of carefully designed capabilities able to get a lot done in just a few lines . Let me show you a couple of examples here . So here 's a trivial piece of Mathematica programming . Here 's something where we 're sort of integrating a bunch of different capabilities here . Here we 'll just create in this line a little user interface that allows us to do something fun there . If you go on , that 's a slightly more complicated program that 's now doing all sorts of algorithmic things and creating user interface and so on . But it 's something that 's very precise stuff . It 's a precise specification with a precise formal language that causes Ma
 thematica to know what to do here . Well , then on the other hand , we have Wolfram Alpha , with all the sort of messiness of the world and human language and so on built into it . So what happens when you put these things together ? I think it 's actually rather wonderful . With Wolfram Alpha inside Mathematica , you can , for example , make precise programs that call on real-world data . Here 's a really simple example . You can also just sort of give vague input and then try and have Wolfram Alpha figure out what you 're talking about . Let 's try this here . But actually I think sort of the most exciting thing about this is that it really gives one the chance to democratize programming . I mean , anyone will be able to just sort of say what they want in plain language , then , the idea is , that Wolfram Alpha will be able to figure out what precise pieces of code can do what they 're asking for and then show them examples that will let them pick what they need to build up bigger
  and bigger , precise programs . So , sometimes , Wolfram Alpha will be able to do the whole thing immediately and just give back a whole big program that you can then compute with . So here 's a big website where we 've been collecting lots of educational and other demonstrations about lots of kinds of things . So , I do n't know , I 'll show you one example , maybe here . This is just an example of one of these computable documents . This is probably a fairly small piece of Mathematica code that 's able to be run here . Okay . Let 's zoom out again . So , given our new kind of science , is there a general way to use it to make technology ? So , with physical materials , we 're used to kind of going around the world and discovering that particular materials are useful for particular technological purposes and so on . Well , it turns out , we can do very much the same kind of thing in the computational universe . There 's an inexhaustible supply of programs out there . The challenge
  is to see how to harness them for human purposes . Something like Rule 30 , for example , turns out to be a really good randomness generator . Other simple programs are good models for processes in the natural or social world . And , for example , Wolfram Alpha and Mathematica are actually now full of algorithms that we discovered by searching the computational universe . And , for example , this -- we go back here -- This has become surprisingly popular among composers finding musical forms by searching the computational universe . In a sense , we can use the computational universe to get mass customized creativity . I 'm hoping we can , for example , use that even to get Wolfram Alpha to routinely sort of do invention and discovery on the fly and to find all sorts of wonderful stuff that no engineer and no process of incremental evolution would ever come up with . Well , so , that leads to sort of an ultimate question . Could it be that someplace out there in the computational un
 iverse we might find our physical universe ? Perhaps there 's even some quite simple rule , some simple program for our universe . Well , the history of physics would have us believe that the rule for the universe must be pretty complicated . But in the computational universe we 've now seen how rules that are incredibly simple can produce incredibly rich and complex behavior . So could that be what 's going on with our whole universe ? If the rules for the universe are simple , it 's kind of inevitable that they have to be very abstract and very low level , operating , for example , far below the level of space or time , which makes it hard to represent things . But in at least a large class of cases , one can think of the universe as being like some kind of network , which , when it gets big enough , behaves like continuous space in much the same way as having lots of molecules can behave like a continuous fluid . Well , then the universe has to evolve by applying little rules tha
 t progressively update this network . And each possible rule , in a sense , corresponds to a candidate universe . Actually , I have n't shown these before , but here are a few of the candidate universes that I 've looked at . Some of these are hopeless universes , completely sterile , with other kinds of pathologies like no notion of space , no notion of time , no matter , other problems like that . But the exciting thing that I 've found in the last few years is that you actually do n't have to go very far in the computational universe before you start finding candidate universes that are n't obviously not our universe . Here 's the problem : Any serious candidate for our universe , is inevitably full of computational irreducibility , which means that it is irreducibly difficult to find out how it will really behave , and whether it matches our physical universe . A few years ago , I was pretty excited to discover that there are candidate universes with incredibly simple rules that
  successfully reproduce special relativity and even general relativity and gravitation and at least give hints of quantum mechanics . So , will we find the whole of physics ? I do n't know for sure . But I think at this point it 's sort of almost embarrassing not to at least try . Not an easy project . One has got to build a lot of technology . One 's got to build a structure that 's probably at least as deep as existing physics . And I 'm not sure what the best way to organize the whole thing is . Build a team , open it up , offer prizes and so on . But I 'll tell you here today that I 'm committed to seeing this project done , to see if , within this decade , we can finally hold in our hands the rule for our universe and know where our universe lies in the space of all possible universes -- and be able to type into Wolfram Alpha " the theory of the universe , " and have it tell us . ( Laughter ) So I 've been working on the idea of computation now for more than 30 years , building
  tools and methods and turning sort of intellectual ideas into millions of lines of code and grist for server farms and so on . With every passing year , I realize how much more powerful the idea of computation really is . It 's taken us a long way already , but there 's so much more to come . From the foundations of science to the limits of technology to the very definition of the human condition , I think computation is destined to be the defining idea of our future . Thank you . ( Applause ) Chris Anderson : That was astonishing . Stay here . I 've got a question . ( Applause ) So , that was , fair to say , an astonishing talk . Are you able to say in a sentence or two how this type of thinking could integrate at some point to things like string theory or the kind of things that people think of as the fundamental explanations of the universe ? Stephen Wolfram : Well , the parts of physics that we kind of know to be true , things like the standard model of physics . What I 'm tryi
 ng to do better reproduce the standard model of physics or it 's simply wrong . The things that people have tried to do in the last 25 years or so with string theory and so on have been an interesting exploration that has tried to get back to the standard model , but has n't quite gotten there . My guess is that some great simplifications of what I 'm doing may actually have considerable resonance with what 's been done in string theory , but that 's a complicated math thing that I do n't yet know how it 's going to work out . CA : Benoit Mandlebrot is in the audience . He has also shown how complexity can arise from a simple start . Does your work relate to his ? SW : I think so . I view Benoit Mandlebrot 's work as kind of one of the founding contributions to this kind of area . Benoit has been particularly interested in nested patterns , in fractals and so on , where the structure is something that 's kind of tree-like , and where there 's sort of a big branch that makes little b
 ranches , and even smaller branches and so on . That 's kind of one of the ways that you get towards true complexity . I think things like the Rule 30 cellular automaton get us to a different level . In fact , in a very precise way they get us to a different level because they seem to be things that are capable of complexity that 's sort of as great as complexity can ever get ... I could go on about this at great length , but I wo n't . CA : Stephen Wolfram , thank you . ( Applause ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/175ted_tom_wujec_build_a_tower.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/175ted_tom_wujec_build_a_tower.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/175ted_tom_wujec_build_a_tower.txt
new file mode 100644
index 0000000..ad09734
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/175ted_tom_wujec_build_a_tower.txt
@@ -0,0 +1,2 @@
+
+Several years ago , here at TED , Peter Skillman introduced a design challenge called the marshmallow challenge . And the idea 's pretty simple . Teams of four have to build the tallest free-standing structure out of 20 sticks of spaghetti , one yard of tape , one yard of string and a marshmallow . The marshmallow has to be on top . And , though it seems really simple , it 's actually pretty hard , because it forces people to collaborate very quickly . And so I thought that this was an interesting idea , and I incorporated it into a design workshop . And it was a huge success . And since then , I 've conducted about 70 design workshops across the world with students and designers and architects , even the CTOs of the Fortune 50 , and there 's something about this exercise that reveals very deep lessons about the nature of collaboration , and I 'd like to share some of them with you . So , normally , most people begin by orienting themselves to the task . They talk about it , they fi
 gure out what it 's going to look like , they jockey for power , then they spend some time planning , organizing . They sketch and they lay out spaghetti They spend the majority of their time assembling the sticks into ever-growing structures and then , finally , just as they 're running out of time , someone takes out the marshmallow , and then they gingerly put it on top , and then they stand back , and Ta-da ! they admire their work . But what really happens , most of the time , is that the " ta-da " turns into an " uh-oh , " because the weight of the marshmallow causes the entire structure to buckle and to collapse . So there are a number of people who have a lot more " uh-oh " moments than others , and among the worst are recent graduates of business school . ( Laughter ) They lie , they cheat , they get distracted , and they produce really lame structures . And of course there are teams that have a lot more " ta-da " structures , and , among the best , are recent graduates of 
 kindergarten . ( Laughter ) And it 's pretty amazing . As Peter tells us , not only do they produce the tallest structures , but they 're the most interesting structures of them all . So the question you want to ask is : How come ? Why ? What is it about them ? And Peter likes to say that , " None of the kids spend any time trying to be CEO of Spaghetti Inc. " Right . They do n't spend time jockeying for power . But there 's another reason as well . And the reason is that business students are trained to find the single right plan , right . And then they execute on it . And then what happens is , when they put the marshmallow on the top , they run out of time , and what happens ? It 's a crisis . Sound familiar ? Right . What kindergarteners do differently , is that they start with the marshmallow , and they build prototypes , successive prototypes , always keeping the marshmallow on top , so they have multiple times to fix ill built prototypes along the way . So designers recognize
  this type of collaboration as the essence of the iterative process . And with each version , kids get instant feedback about what works and what does n't work . So the capacity to play in prototype is really essential , but let 's look at how different teams perform . So the average for most people is around 20 inches , business schools students , about half of that , lawyers , a little better , but not much better than that , kindergarteners , better than most adults . Who does the very best ? Architects and engineers , thankfully . ( Laughter ) 39 inches is the tallest structure I 've seen . And why is it ? Because they understand triangles and self-re-enforcing geometrical patterns are the key to building stable structures . So CEOs , a little bit better than average . But here 's where it gets interesting . If you put you put an executive admin . on the team , they get significantly better . ( Laughter ) It 's incredible . You know , you look around , you go , " Oh , that team 
 's going to win . " You can just tell beforehand . And why is that ? Because they have special skills of facilitation . They manage the process , they understand the process . And any team who manages and pays a close attention to work will significantly improve the team 's performance . Specialized skills and facilitation skills are the combination [ that ] leads to strong success . If you have 10 teams that typically perform , you 'll get maybe six or so that have standing structures . And I tried something interesting . I thought , let 's up the ante once . So I offered a 10,000 dollar prize of software to the winning team . So what do you think happened to these design students ? What was the result ? Here 's what happened . Not one team had a standing structure . If anyone had built , say , a one inch structure , they could have taken home the prize . So , is n't it interesting that high stakes have a strong impact . We did the exercise again with the same students . What do yo
 u think happened then ? So now they understand the value of prototyping . So the same team went from being the very worst to being among the very best . They produced the tallest structures in the least amount of time . So there 's deep lessons for us about the nature of incentives and success . So , you might ask : Why would anyone actually spend time writing a marshmallow challenge ? And the reason is , I help create digital tools and processes to help teams build cars and video games and visual effects . And what the marshmallow challenge does is it helps them identify the hidden assumptions . Because , frankly , every project has its own marshmallow , does n't it . The challenge provides a shared experience , a common language , common stance to build the right prototype . And so , this is the value of the experience , of this so simple exercise . And those of you who are interested , may want to go to marshmallowchallenge . com . It 's a blog that you can look at how to build t
 he marshmallows . There 's step-by-step instructions on this . There are crazy examples from around the world of how people tweak and adjust the system . There 's world records on this as well . And the fundamental lesson , I believe , is that design truly is a contact sport . It demands that we bring all of our senses to the task , and that we apply the very best of our thinking , our feeling and our doing to the challenge that we have at hand . And , sometimes , a little prototype of this experience is all that it takes to turn us from an " uh-oh " moment to a " ta-da " moment . And that can make a big difference . Thank you very much . ( Applause ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/176ted_tom_wujec_on_3_ways_the_brain_creates_meaning.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/176ted_tom_wujec_on_3_ways_the_brain_creates_meaning.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/176ted_tom_wujec_on_3_ways_the_brain_creates_meaning.txt
new file mode 100644
index 0000000..44a13c7
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/ted/176ted_tom_wujec_on_3_ways_the_brain_creates_meaning.txt
@@ -0,0 +1,2 @@
+
+Last year at TED we aimed to try to clarify the overwhelming complexity and richness that we experience at the conference in a project called Big Viz . And the Big Viz is a collection of 650 sketches that were made by two visual artists . David Sibbet from The Grove , and Kevin Richards from Autodesk made 650 sketches that strive to capture the essence of each presenter 's ideas . And the consensus was , it really worked . These sketches brought to life the key ideas , the portraits , the magic moments that we all experienced last year . This year we were thinking , " Why does it work ? " What is it about animation , graphics , illustrations , that create meaning ? And this is an important question to ask and answer because the more we understand how the brain creates meaning , the better we can communicate , and I also think , the better we can think and collaborate together . So this year we 're going to visualize how the brain visualizes . Cognitive psychologists now tell us that
  the brain does n't actually see the world as it is , but instead , creates a series of mental models through a collection of " Ah-ha moments , " or moments of discovery , through various processes . The processing , of course , begins with the eyes . Light enters , hits the back of the retina , and is circulated , most of which is streamed to the very back of the brain , at the primary visual cortex . And primary visual cortex sees just simple geometry , just the simplest of shapes . But it also acts like a kind of relay station that re-radiates and redirects information to many other parts of the brain . As many as 30 other parts that selectively make more sense , create more meaning through the kind of " Ah-ha " experiences . We 're only going to talk about three of them . So the first one is called the ventral stream . It 's on this side of the brain . And this is the part of the brain that will recognize what something is . It 's the " what " detector . Look at a hand . Look at
  a remote control . Chair . Book . So that 's the part of the brain that is activated when you give a word to something . A second part of the brain is called the dorsal stream . And what it does is locates the object in physical body space . So if you look around the stage here you 'll create a kind of mental map of the stage . And if you closed your eyes you 'd be able to mentally navigate it . You 'd be activating the dorsal stream if you did that . The third part that I 'd like to talk about is the limbic system . And this is deep inside of the brain . It 's very old , evolutionarily . And it 's the part that feels . It 's the kind of gut center , where you see an image and you go , " Oh ! I have a strong or emotional reaction to whatever I 'm seeing . " So the combination of these processing centers help us make meaning in very different ways . So what can we learn about this ? How can we apply this insight ? Well , again , the schematic view is that the eye visually interrogat
 es what we look at . The brain processes this in parallel , the figments of information asking a whole bunch of questions to create a unified mental model . So , for example , when you look at this image a good graphic invites the eye to dart around , to selectively create a visual logic . So the act of engaging , and looking at the image creates the meaning . It 's the selective logic . Now we 've augmented this and spatialized this information . Many of you may remember the magic wall that we built in conjunction with Perceptive Pixel where we quite literally create an infinite wall . And so we can compare and contrast the big ideas . So the act of engaging and creating interactive imagery enriches meaning . It activates a different part of the brain . And then the limbic system is activated when we see motion , when we see color . and there are primary shapes and pattern detectors that we 've heard about before . So the point of this is what ? We make meaning by seeing , by an ac
 t of visual interrogation . The lessons for us are three-fold . First , use images to clarify what we 're trying to communicate . Secondly make those images interactive so that we engage much more fully . And the third is to augment memory by creating a visual persistence . These are techniques that can be used to be -- that can be applied in a wide range of problem solving . So the low-tech version looks like this . And , by the way , this is the way in which we develop and formulate strategy within Autodesk , in some of our organizations and some of our divisions . What we literally do is have the teams draw out the entire strategic plan on one giant wall . And it 's very powerful because everyone gets to see everything else . There 's always a room , always a place to be able to make sense of all of the components in the strategic plan . This is a time-lapse view of it . You can ask the question , " Who 's the boss ? " You 'll be able to figure that out . So the act of collective
 ly and collaboratively building the image transforms the collaboration . No Powerpoint is used in two days. But instead the entire team creates a shared mental model that they can all agree on and move forward on . And this can be enhanced and augmented with some emerging digital technology . And this is our great unveiling for today . And this is an emerging set of technologies that use large-screen displays with intelligent calculation in the background to make the invisible visible . Here what we can do is look at sustainability , quite literally . So a team can actually look at all the key components that heat the structure and make choices and then see the end result that is visualized on this screen . So making images meaningful has three components . The first again , is making ideas clear by visualizing them . Secondly , making them interactive . And then thirdly , making them persistent . And I believe that these three principles can be applied to solving some of the very t
 ough problems that we face in the world today . Thanks so much . ( Applause ) 
\ No newline at end of file


[23/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/96TediJordanC_ShockStats_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/96TediJordanC_ShockStats_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/96TediJordanC_ShockStats_EN.txt.txt
new file mode 100644
index 0000000..38ab77c
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/96TediJordanC_ShockStats_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff My work is about the behaviors that we all engage in unconsciously , on a collective level . And what I mean by that , it 's the behaviors that we 're in denial about , and the ones that operate below the surface of our daily awareness . And as individuals , we all do these things , all the time , everyday . It 's like when you 're mean to your wife because you 're mad at somebody else . Or when you drink a little too much at a party just out of anxiety . Or when you overeat because your feelings are hurt , or whatever . And when we do these kind of things , when 300 million people do unconscious behaviors , then it can add up to a catastrophic consequence that nobody wants , and no one intended . And that 's what I look at with my photographic work . This is an image I just recently completed , that is , when you stand back at a distance , it looks like some kind of neo-gothic , cartoon image of a factory spewing out pollution . And as you get a little bit closer , it starts lo
 oking like lots of pipes , like maybe a chemical plant , or a refinery , or maybe a hellish freeway interchange . And as you get all the way up close , you realize that it 's actually made of lots and lots of plastic cups . And in fact , this is one million plastic cups , which is the number of plastic cups that are used on airline flights in the United States every six hours . We use four million cups a day on airline flights , and virtually none of them are reused or recycled ; they just do n't do that in that industry . Now that number is dwarfed by the number of paper cups we use every day , and that is 40 million cups a day for hot beverages , most of which is coffee . I could n't fit 40 million cups on a canvas , but I was able to put 410,000 . That 's what 410,000 cups looks like . That 's 15 minutes of our cup consumption . And if you could actually stack up that many cups in real life , that 's the size it would be . And there 's an hour 's worth of our cups . And there 's 
 a day 's worth of our cups . You can still see the little people way down there . That 's as high as a 42-story building , and I put the Statue of Liberty in there as a scale reference . Speaking of justice , there 's another phenomenon going on in our culture that I find deeply troubling , and that is that America right now has the largest percentage of its population in prison of any country on Earth . One out of four people , one out of four humans in prison are Americans , imprisoned in our country . And I wanted to show the number . The number is 2.3 million Americans were incarcerated in 2005. And that 's gone up since then , but we do n't have the numbers yet . So I wanted to show 2.3 million prison uniforms , and in the actual print of this piece , each uniform is the size of a nickel on its edge . They 're tiny , they 're barely visible as a piece of material , and to show 2.3 million of them required a canvas that was larger than any printer in the world would print . And 
 so I had to divide it up into multiple panels that are 10 feet tall by 25 feet wide . This is that piece installed in a gallery in New York ; those are my parents looking at the piece . ( Laughter ) Every time I look at this piece , I always wonder if my mom 's whispering to my dad , " He finally folded his laundry . " ( Laughter ) I want to show you some pieces now that are about addiction . And this particular one is about cigarette addiction . I wanted to make a piece that shows the actual number of Americans who die from cigarette smoking . More than 400,000 people die in the United States every year from smoking cigarettes . And so this piece is made up of lots and lots of boxes of cigarettes . And , as you slowly step back , you see that it 's a painting by Van Gogh , called " Skull with Cigarette . " It 's a strange thing to think about , that on 9/11 , when that tragedy happened , 3,000 Americans died , and do you remember the response ? It reverberated around the world , an
 d will continue to reverberate through time . It will be something that we talk about in 100 years . And yet on that same day , 1,100 Americans died from smoking . And the day after that , another 1,100 Americans died from smoking . And every single day since then , 1,100 Americans have died , and today , 1,100 Americans are dying from cigarette smoking . And we are n't talking about it ; we dismiss it . The tobacco lobby , it 's too strong . We just dismiss it out of our consciousness . And knowing what we know about the destructive power of cigarettes , we continue to allow our children , our sons and daughters , to be in the presence of the influences that start them smoking . And this is what the next piece is about . This is just lots and lots of cigarettes : 65,000 cigarettes , which is equal to the number of teenagers who will start smoking this month , and every month in the US . More than 700,000 children in the United States aged 18 and under begin smoking every year . One
  more strange epidemic in the United States that I want to acquaint you with is this phenomenon of abuse and misuse of prescription drugs . This is an image I 've made out of lots and lots of Vicodin -- well , actually I only had one Vicodin that I scanned lots and lots of times . ( Laughter ) And so as you stand back you see 213,000 Vicodin pills , which is the number of hospital emergency room visits yearly in the United States , attributable to abuse and misuse of prescription painkillers and anti-anxiety medications . One-third of all drug overdoses in the U. S. -- and that includes cocaine , heroin , alcohol , everything -- one-third of drug overdoses are prescription medications . A strange phenomenon . This is a piece that I just recently completed about another tragic phenomenon . And that is the phenomenon , this growing obsession we have with breast augmentation surgery . 384,000 women , American women , last year went in for elective breast augmentation surgery . It 's ra
 pidly becoming the most popular high school graduation gift , given to young girls who are about to go off to college . So I made this image out of Barbie dolls , and so as you stand back you see this kind of floral pattern , and as you get all the way back , you see 32,000 Barbie dolls , which represents the number of breast augmentation surgeries that are performed in the U. S. each month . The vast majority of those are on women under the age of 21. And strangely enough , the only plastic surgery that is more popular than breast augmentation is liposuction , and most of that is being done by men . Now , I want to emphasize that these are just examples . I 'm not holding these out as being the biggest issues . They 're just examples . And the reason that I do this , it 's because I have this fear that we are n't feeling enough as a culture right now . There 's this kind of anesthesia in America at the moment . We 've lost our sense of outrage , our anger and our grief about what '
 s going on in our culture right now , what 's going on in our country , the atrocities that are being committed in our names around the world . They 've gone missing ; these feelings have gone missing . Our cultural joy , our national joy is nowhere to be seen . And one of the causes of this , I think , is that as each of us attempts to build this new kind of world view , this holoptical world view , this holographic image that we 're all trying to create in our mind of the inter-connection of things : the environmental footprints 1,000 miles away of the things that we buy ; the social consequences 10,000 miles away of the daily decisions that we make as consumers . As we try to build this view , and try to educate ourselves about the enormity of our culture , the information that we have to work with is these gigantic numbers : numbers in the millions , in the hundreds of millions , in the billions and now in the trillions . Bush 's new budget is in the trillions , and these are nu
 mbers that our brain just does n't have the ability to comprehend . We ca n't make meaning out of these enormous statistics . And so that 's what I 'm trying to do with my work , is to take these numbers , these statistics from the raw language of data , and to translate them into a more universal visual language , that can be felt . Because my belief is , if we can feel these issues , if we can feel these things more deeply , then they 'll matter to us more than they do now . And if we can find that , then we 'll be able to find within each one of us what it is that we need to find to face the big question , which is : How do we change ? That , to me , is the big question that we face as a people right now : How do we change ? How do we change as a culture , and how do we each individually take responsibility for the one piece of the solution that we are in charge of , and that is our own behavior ? My belief is that you do n't have to make yourself bad to look at these issues . I 
 'm not pointing the finger at America in a blaming way . I 'm simply saying , this is who we are right now . And if there are things that we see that we do n't like about our culture , then we have a choice . The degree of integrity that each of us can bring to the surface , to bring to this question , the depth of character that we can summon as we show up for the question of how do we change . It 's already defining us as individuals and as a nation , and it will continue to do that on into the future . And it will profoundly affect the well-being , the quality of life , of the billions of people who are going to inherit the results of our decisions . I 'm not speaking abstractly about this , I 'm speaking -- this is who we are in this room . Right now in this moment . Thank you and good afternoon . ( Applause ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/97TediMcKeanE_Dict_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/97TediMcKeanE_Dict_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/97TediMcKeanE_Dict_EN.txt.txt
new file mode 100644
index 0000000..a868bf0
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/97TediMcKeanE_Dict_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Now , have any of y'all ever looked up this word ? You know , in a dictionary ? ( Laughter ) Yeah , that 's what I thought . How about this word ? Here , I 'll show it to you : Lexicography : the practice of compiling dictionaries . Notice -- we 're very specific . That word " compile . " The dictionary is not carved out of a piece of granite , out of a lump of rock . It 's made up of lots of little bits . It 's little discrete -- that 's spelled D-I-S-C-R-E-T-E -- bits . And those bits are words . Now one of the perks of being a lexicographer -- besides getting to come to TED -- is that you get to say really fun words , like lexicographical . Lexicographical has this great pattern -- it 's called a double dactyl . And just by saying double dactyl , I 've sent the geek needle all the way into the red . But " lexicographical " is the same pattern as " higgledy-piggledy . " Right ? It 's a fun word to say , and I get to say it a lot . Now , one of the non-perks of being a lexicogr
 apher is that people do n't usually have a kind of warm , fuzzy , snuggly image of the dictionary . Right ? Nobody hugs their dictionaries . But what people really often think about the dictionary is , they think more like this . Just to let you know , I do not have a lexicographical whistle . But people think that my job is to let the good words make that difficult left hand turn into the dictionary , and keep the bad words out . But the thing is , I do n't want to be a traffic cop . For one thing , I just do not do uniforms . And for another -- deciding what words are good and what words are bad is actually not very easy . And it 's not very fun . And when parts of your job are not easy or fun , you kind of look for an excuse not to do them . So if I had to think of some kind of occupation as a metaphor for my work , I would much rather be a fisherman . I wanna throw my big net into the deep blue ocean of English and see what marvelous creatures I can drag up from the bottom . But
  why do people want me to direct traffic , when I would much rather go fishing ? Well , I blame the Queen . Why do I blame the Queen ? Well , first of all , I blame the Queen because it 's funny . But secondly , I blame the Queen because dictionaries have really not changed . Our idea of what a dictionary is has not changed since her reign . The only thing Queen Victoria would not be amused by in modern dictionaries is our inclusion of the F-word , which has happened in American dictionaries since 1965. So , there 's this guy , right ? Victorian era . James Murray , first editor of the Oxford English Dictionary . I do not have that hat . I wish I had that hat . So he 's really responsible for a lot of what we consider modern in dictionaries today . When a guy who looks like that -- in that hat -- is the face of modernity , you have a problem . And so , James Murray could get a job on any dictionary today . There 'd be virtually no learning curve . And of course , a few of us are say
 ing : Computers ! Computers ! What about computers ? The thing about computers is -- I love computers . I mean , I 'm a huge geek , I love computers . I would go on a hunger strike before I let them take away Google Book Search from me . But computers do n't do much else other than speed up the process of compiling dictionaries . They do n't change the end result . Because what a dictionary is , is it 's Victorian design merged with a little bit of modern propulsion . It 's steampunk . What we have is an electric velocipede . You know , we have Victorian design with an engine on it . That 's all ! The design has not changed . And OK , what about online dictionaries , right ? Online dictionaries must be different . This is the Oxford English Dictionary Online , one of the best online dictionaries . This is my favorite word , by the way : Erinaceous : Pertaining to the hedgehog family ; of the nature of a hedgehog . Very useful word . So look at that . Online dictionaries right now ar
 e paper thrown up on a screen . This is flat . Look how many links there are in the actual entry : two ! Right ? Those little buttons -- I had them all expanded except for the date chart . So there 's not very much going on here . There 's not a lot of clickiness . And in fact , online dictionaries replicate almost all the problems of print , except for searchability . And when you improve searchability , you actually take away the one advantage of print , which is serendipity . Serendipity is when you find things you were n't looking for because finding what you are looking for is so damned difficult . So -- ( Laughter ) -- now , when you think about this , what we have here is a ham butt problem . Does everyone know the ham butt problem ? Woman 's making a ham for a big family dinner . She goes to cut the butt off the ham and throw it away , and she looks at this piece of ham and she 's like , " This is a perfectly good piece of ham . Why am I throwing this away ? " She thought , 
 " Well my mom always did this . " So she calls up Mom , and she says , " Mom , why 'd you cut the butt off the ham when you 're making a ham ? " She says , " I do n't know , my mom always did it ! " So they call Grandma , and Grandma says , " My pan was too small ! " ( Laughter ) So it 's not that we have good words and bad words -- we have a pan that 's too small ! You know , that ham butt is delicious ! There 's no reason to throw it away . The bad words -- see , when people think about a place and they do n't find a place on the map , they think , " This map sucks ! " When they find a nightspot or a bar and it 's not in the guidebook , they 're like , " Ooh , this place must be cool ! It 's not in the guidebook . " When they find a word that 's not in the dictionary , they think , " This must be a bad word . " Why ? It 's more likely to be a bad dictionary . Why are you blaming the ham for being too big for the pan ? So you ca n't get a smaller ham . The English language is as bi
 g as it is . So if you have a ham butt problem , and you 're thinking about the ham butt problem , the conclusion it leads you to is inexorable and counter-intuitive : paper is the enemy of words . How can this be ? I mean , I love books . I really love books . Some of my best friends are books . But the book is not the best shape for the dictionary . Now they 're gonna think " Oh , boy . People are gonna take away my beautiful , paper dictionaries ? " No. There will still be paper dictionaries . When we had cars -- when cars became the dominant mode of transportation , we did n't round up all the horses and shoot them . You know , there 're still gonna be paper dictionaries , but it 's not going to be the dominant dictionary . The book-shaped dictionary is not going to be the only shape dictionaries come in . And it 's not going to be the prototype for the shapes dictionaries come in . So think about it this way : if you 've got an artificial constraint , artificial constraints lea
 d to arbitrary distinctions and a skewed worldview . What if biologists could only study animals that made people go , " Aww . " Right ? What if we made aesthetic judgments about animals , and only the ones we thought were cute were the ones that we could study ? We 'd know a whole lot about charismatic megafauna , and not very much about much else . And I think this is a problem . I think we should study all the words , because when you think about words , you can make beautiful expressions from very humble parts . Lexicography is really more about material science . We are studying the tolerances of the materials that you use to build the structure of your expression : your speeches and your writing . And then often people say to me , " Well , OK -- how do I know that this word is real ? " They think , " OK , if we think words are the tools that we use to build the expressions of our thoughts , how can you say that screwdrivers are better than hammers ? How can you say that a sled
 gehammer is better than a ball-peen hammer ? They 're just the right tool for the job . " And so people say to me , " How do I know if a word is real ? " You know , anyone that 's read a children 's book knows that love makes things real . If you love a word , use it . That makes it real . Being in the dictionary is an artificial distinction . It does n't make a word any more real than any other way . If you love a word , it becomes real . So if we 're not worrying about directing traffic , if we 've transcended paper , if we are worrying less about control and more about description , then we can think of the English language as being this beautiful mobile . And any time one of those little parts of the mobile changes , is touched -- any time you touch a word , you use it in a new context , you give it a new connotation , you verb it -- you make the mobile move . You did n't break it ; it 's just in a new position , and that new position can be just as beautiful . Now , if you 're 
 no longer a traffic cop -- the problem with being a traffic cop is there can only be so many traffic cops in any one intersection , or the cars get confused . Right ? But if your goal is no longer to direct the traffic , but maybe to count the cars that go by , then more eyeballs are better . You can ask for help ! If you ask for help , you get more done . And we really need help . Library of Congress : 17 million books . Of which half are in English . If only one out of every 10 of those books had a word that 's not in the dictionary in it , that would be equivalent to more than two unabridged dictionaries . And I find an un-dictionaried word -- a word like " un-dictionaried , " for example -- in almost every book I read . What about newspapers ? Newspaper archive goes back to 1759. 58.1 million newspaper pages . If only one in 100 of those pages had an un-dictionaried word on it , it would be an entire other OED . That 's 500,000 more words . So that 's -- that 's a lot . And I 'm
  not even talking about magazines , I 'm not talking about blogs -- and I find more new words on BoingBoing in a given week than I do Newsweek or Time . There 's a lot going on there . And I 'm not even talking about polysemy , which is the greedy habit some words have of taking more than one meaning for themselves . So if you think of the word " set " -- a set can be a badger 's burrow , a set can be one of the pleats in an Elizabethan ruff -- and there 's one numbered definition in the OED . The OED has 33 different numbered definitions for set . Tiny little word , 33 numbered definitions . One of them is just labeled " miscellaneous technical senses . " Do you know what that says to me ? That says to me it was Friday afternoon and somebody wanted to go down the pub . That 's a lexicographical cop out , to say , " miscellaneous technical senses . " So we have all these words , and we really need help ! And the thing is , we could ask for help -- asking for help 's not that hard . 
 I mean , lexicography is not rocket science . See , I just gave you a lot of words and a lot of numbers , and this is more of a visual explanation . If we think of the dictionary as being the map of the English language , these bright spots are what we know about and the dark spots are where we are in the dark . If that was the map of all the words in American English , we do n't know very much . And we do n't even know the shape of the language . If this was the dictionary -- if this was the map of American English -- look , we have a kind of lumpy idea of Florida , but there 's no California ! We 're missing California from American English . We just do n't know enough , and we do n't even know we 're missing California . We do n't even see that there 's a gap on the map . So again , lexicography is not rocket science . But even if it were , rocket science is being done by dedicated amateurs these days. You know ? It ca n't be that hard to find some words ! So , enough scientists 
 in other disciplines are really asking people to help , and they 're doing a good job of it . For instance : there 's eBird , where amateur birdwatchers can upload information about their bird sightings . And then ornithologists can go and help track populations , migrations , et cetera . And there 's this guy Mike Oates . Mike Oates lives in the U. K. He 's a director of an electroplating company . He 's found more than 140 comets . He 's found so many comets , they named a comet after him . It 's kind of out past Mars -- it 's a hike . I do n't think he 's getting his picture taken there anytime soon . But he found 140 comets without a telescope . He downloaded data from the NASA SOHO satellite , and that 's how he found them . If we can find comets without a telescope , should n't we be able to find words ? Now , y'all know where I 'm going with this . Because I 'm going to the Internet , which is where everybody goes . And the Internet is great for collecting words , because the
  Internet 's full of collectors . And this is a little-known technological fact about the Internet , but the Internet is actually made up of words and enthusiasm . And words and enthusiasm actually happen to be the recipe for lexicography . Is n't that great ? So there are a lot of really good word-collecting sites out there right now , but the problem with some of them is that they 're not scientific enough . They show the word , but they do n't show any context : Where did it come from ? Who said it ? What newspaper was it in ? What book ? Because a word is like an archaeological artifact . If you do n't know the provenance or the source of the artifact , it 's not science -- it 's a pretty thing to look at . So a word without its source is like a cut flower . You know -- it 's pretty to look at for a while , but then it dies . It dies too fast . So this whole time I 've been saying , " The dictionary , the dictionary , the dictionary , the dictionary . " Not " a dictionary , " or
  " dictionaries . " And that 's because -- well , people use the dictionary to stand for the whole language . They use it synecdochically -- and one of the problems of knowing a word like " synecdochically " is that you really want an excuse to say synecdochically . This whole talk has just been an excuse to get me to the point where I could say synecdochically to all of you . So I 'm really sorry . But when you use a part of something -- like the dictionary is a part of the language , or a flag stands for the United States , a symbol of the country -- then you 're using it synecdochically . But the thing is , we could make the dictionary the whole language . If we get a bigger pan , then we can put all the words in . We can put in all the meanings . Does n't everyone want more meaning in their lives ? And we can make the dictionary not just be a symbol of the language -- we can make it be the whole language . You see , what I 'm really hoping for is that my son -- who turns seven t
 his month -- I want him to barely remember that this is the form factor that dictionaries used to come in . This is what dictionaries used to look like . I want him to think of this kind of dictionary as an eight-track tape . It 's a format that died because it was n't useful enough . It was n't really what people needed . And the thing is , if we can put in all the words , no longer have that artificial distinction between good and bad , we can really describe the language like scientists . We can leave the aesthetic judgments to the writers and the speakers . If we can do that , then I can spend all my time fishing and I do n't have to be a traffic cop anymore . Thank you very much for your kind attention . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/98TediMillerA_News_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/98TediMillerA_News_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/98TediMillerA_News_EN.txt.txt
new file mode 100644
index 0000000..69aec09
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/98TediMillerA_News_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff How does the news shape the way we see the world ? Here 's the world based on the way it looks -- based on land mass . And here 's how news shapes what Americans see . This map -- ( Applause ) -- this map shows the number of seconds that American network and cable news organizations dedicated to news stories , by country , in February of 2007 -- just one year ago . Now , this was a month when North Korea agreed to dismantle its nuclear facilities . There was massive flooding in Indonesia . And in Paris , the IPCC released its study confirming man 's impact on global warming . The U. S. accounted for 79 percent of total news coverage . And when we take out the U. S. and look at the remaining 21 percent , we see a lot of Iraq -- that 's that big green thing there -- and little else . The combined coverage of Russia , China and India , for example , reached just one percent . When we analyzed all the news stories and removed just one story , here 's how the world looked . What was 
 that story ? The death of Anna Nicole Smith . This story eclipsed every country except Iraq , and received 10 times the coverage of the IPCC report . And the cycle continues ; as we all know , Britney has loomed pretty large lately . So why do n't we hear more about the world ? One reason is that news networks have reduced the number of their foreign bureaus by half . Aside from one-person ABC mini-bureaus in Nairobi , New Delhi and Mumbai , there are no network news bureaus in all of Africa , India or South America -- places that are home to more than two billion people . The reality is that covering Britney is cheaper . And this lack of global coverage is all the more disturbing when we see where people go for news . Local TV news looms large , and unfortunately only dedicates 12 percent of its coverage to international news . And what about the web ? The most popular news sites do n't do much better . Last year , Pew and the Colombia J-School analyzed the 14,000 stories that appe
 ared on Google News ' front page . And they , in fact , covered the same 24 news events . Similarly , a study in e-content showed that much of global news from U. S. news creators is recycled stories from the AP wire services and Reuters , and do n't put things into a context that people can understand their connection to it . So , if you put it all together , this could help explain why today 's college graduates as well as less educated Americans know less about the world than their counterparts did 20 years ago . And if you think it 's simply because we are not interested , you would be wrong . In recent years , Americans who say they closely follow global news most of the time grew to over 50 percent . The real question : is this distorted world view what we want for Americans in our increasingly interconnected world ? I know we can do better . And can we afford not to ? Thank you . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/99TediNatchweyJ_WarPhotos_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/99TediNatchweyJ_WarPhotos_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/99TediNatchweyJ_WarPhotos_EN.txt.txt
new file mode 100644
index 0000000..56e473d
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tedi/99TediNatchweyJ_WarPhotos_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff As someone who has spent his entire career trying to be invisible , standing in front of an audience is a cross between an out-of-body experience and a deer caught in the headlights , so please forgive me for violating one of the TED commandments by relying on words on paper , and I only hope I 'm not struck by lightning bolts before I 'm done . I 'd like to begin by talking about some of the ideas that motivated me to become a documentary photographer . I was a student in the '60s , a time of social upheaval and questioning , and on a personal level , an awakening sense of idealism . The war in Vietnam was raging , the Civil Rights Movement was under way , and pictures had a powerful influence on me . Our political and military leaders were telling us one thing , and photographers were telling us another . I believed the photographers , and so did millions of other Americans . Their images fueled resistance to the war and to racism . They not only recorded history , they helped
  change the course of history . Their pictures became part of our collective consciousness and , as consciousness evolved into a shared sense of conscience , change became not only possible , but inevitable . I saw that the free flow of information represented by journalism , specifically visual journalism , can bring into focus both the benefits and the cost of political policies . It can give credit to sound decision making , adding momentum to success . In the face of poor political judgment or political inaction , it becomes a kind of intervention , assessing the damage and asking us to reassess our behavior . It puts a human face on issues which from afar can appear abstract or ideological or monumental in their global impact . What happens at ground level , far from the halls of power , happens to ordinary citizens one by one . And I understood that documentary photography has the ability to interpret events from their point of view . It gives a voice to those who otherwise wo
 uld not have a voice . And as a reaction , it stimulates public opinion and gives impetus to public debate , thereby preventing the interested parties from totally controlling the agenda , much as they would like to . Coming of age in those days made real the concept that the free flow of information is absolutely vital for a free and dynamic society to function properly . The press is certainly a business , and in order to survive it must be a successful business , but the right balance must be found between marketing considerations and journalistic responsibility . Society 's problems ca n't be solved until they 're identified . On a higher plane , the press is a service industry , and the service it provides is awareness . Every story does not have to sell something . There 's also a time to give . That was a tradition I wanted to follow . Seeing the war created such incredibly high stakes for everyone involved and that visual journalism could actually become a factor in conflict
  resolution , I wanted to be a photographer in order to be a war photographer . But I was driven by an inherent sense that a picture that revealed the true face of war would almost by definition be an anti-war photograph . I 'd like to take you on a visual journey through some of the events and issues I 've been involved in over the past 25 years . In 1981 , I went to Northern Ireland . 10 IRA prisoners were in the process of starving themselves to death in protest against conditions in jail . The reaction on the streets was violent confrontation . I saw that the front lines of contemporary wars are not on isolated battlefields , but right where people live . During the early '80s , I spent a lot of time in Central America , which was engulfed by civil wars that straddled the ideological divide of the Cold War . In Guatemala , the central government -- controlled by a oligarchy of European decent -- was waging a scorched earth campaign against an indigenous rebellion , and I saw an 
 image that reflected the history of Latin America : conquest through a combination of the Bible and the sword . An anti-Sandinista guerrilla was mortally wounded as Commander Zero attacked a town in Southern Nicaragua . A destroyed tank belonging to Somoza 's national guard was left as a monument in a park in Managua , and was transformed by the energy and spirit of a child . At the same time , a civil war was taking place in El Salvador , and again , the civilian population was caught up in the conflict . I 've been covering the Palestinian-Israeli conflict since 1981. This is a moment from the beginning of the second intifada , in 2000 , when it was still stones and Molotovs against an army . In 2001 , the uprising escalated into an armed conflict , and one of the major incidents was the destruction of the Palestinian refugee camp in the West Bank town of Jenin . Without the political world to find common ground , the continual friction of tactic and counter-tactic only creates su
 spicion and hatred and vengeance , and perpetuates the cycle of violence . In the '90s , after the breakup of the Soviet Union , Yugoslavia fractured along ethnic fault lines , and civil war broke out between Bosnia , Croatia and Serbia . This is a scene of house-to-house fighting in Mostar , neighbor against neighbor . A bedroom , the place where people share intimacy , where life itself is conceived , became a battlefield . A mosque in northern Bosnia was destroyed by Serbian artillery and was used as a makeshift morgue . Dead Serbian soldiers were collected after a battle and used as barter for the return of prisoners or Bosnian soldiers killed in action . This was once a park . The Bosnian soldier who guided me told me that all of his friends were there now . At the same time in South Africa , after Nelson Mandela had been released from prison , the black population commenced the final phase of liberation from apartheid . One of the things I had to learn as a journalist was what
  to do with my anger . I had to use it , channel its energy , turn it into something that would clarify my vision , instead of clouding it . In Transkei , I witnessed a rite of passage into manhood , of the Xhosa tribe . Teenage boys lived in isolation , their bodies covered with white clay . After several weeks , they washed off the white and took on the full responsibilities of men . It was a very old ritual that seemed symbolic of the political struggle that was changing the face of South Africa . Children in Soweto playing on a trampoline . Elsewhere in Africa there was famine . In Somalia , the central government collapsed and clan warfare broke out . Farmers were driven off their land , and crops and livestock were destroyed or stolen . Starvation was being used as a weapon of mass destruction -- primitive but extremely effective . Hundreds of thousands of people were exterminated , slowly and painfully . The international community responded with massive humanitarian relief ,
  and hundreds of thousands of more lives were saved . American troops were sent to protect the relief shipments , but they were eventually drawn into the conflict , and after the tragic battle in Mogadishu , they were withdrawn . In southern Sudan , another civil war saw similar use of starvation as a means of genocide . Again , international NGOs , united under the umbrella of the UN , staged a massive relief operation and thousands of lives were saved . I 'm a witness , and I want my testimony to be honest and uncensored . I also want it to be powerful and eloquent , and to do as much justice as possible to the experience of the people I 'm photographing . This man was in an NGO feeding center , being helped as much as he could be helped . He literally had nothing . He was a virtual skeleton , yet he could still summon the courage and the will to move . He had not given up , and if he did n't give up , how could anyone in the outside world ever dream of losing hope ? In 1994 , aft
 er three months of covering the South African election , I saw the inauguration of Nelson Mandela , and it was the most uplifting thing I 've ever seen . It exemplified the best that humanity has to offer . The next day I left for Rwanda , and it was like taking the express elevator to hell . This man had just been liberated from a Hutu death camp . He allowed me to photograph him for quite a long time , and he even turned his face toward the light , as if he wanted me to see him better . I think he knew what the scars on his face would say to the rest of the world . This time , maybe confused or discouraged by the military disaster in Somalia , the international community remained silent , and somewhere around 800,000 people were slaughtered by their own countrymen -- sometimes their own neighbors -- using farm implements as weapons . Perhaps because a lesson had been learned by the weak response to the war in Bosnia and the failure in Rwanda , when Serbia attacked Kosovo internati
 onal action was taken much more decisively . NATO forces went in , and the Serbian army withdrew . Ethnic Albanians had been murdered , their farms destroyed and a huge number of people forcibly deported . They were received in refugee camps set up by NGOs in Albania and Macedonia . The imprint of a man who had been burned inside his own home . The image reminded me of a cave painting , and echoed how primitive we still are in so many ways . Between 1995 and '96 , I covered the first two wars in Chechnya from inside Grozny . This is a Chechen rebel on the front line against the Russian army . The Russians bombarded Grozny constantly for weeks , killing mainly the civilians who were still trapped inside . I found a boy from the local orphanage wandering around the front line . My work has evolved from being concerned mainly with war to a focus on critical social issues as well . After the fall of Ceausescu , I went to Romania and discovered a kind of gulag of children , where thousan
 ds of orphans were being kept in medieval conditions . Ceausescu had imposed a quota on the number of children to be produced by each family , thereby making women 's bodies an instrument of state economic policy . Children who could n't be supported by their families were raised in government orphanages . Children with birth defects were labeled incurables , and confined for life to inhuman conditions . As reports began to surface , again international aid went in . Going deeper into the legacy of the Eastern European regimes , I worked for several months on a story about the effects of industrial pollution , where there had been no regard for the environment or the health of either workers or the general population . An aluminum factory in Czechoslovakia was filled with carcinogenic smoke and dust , and four out of five workers came down with cancer . After the fall of Suharto in Indonesia , I began to explore conditions of poverty in a country that was on its way towards moderniz
 ation . I spent a good deal of time with a man who lived with his family on a railway embankment and had lost an arm and a leg in a train accident . When the story was published , unsolicited donations poured in . A trust fund was established , and the family now lives in a house in the countryside and all their basic necessities are taken care of . It was a story that was n't trying to sell anything . Journalism had provided a channel for people 's natural sense of generosity , and the readers responded . I met a band of homeless children who 'd come to Jakarta from the countryside , and ended up living in a train station . By the age of 12 or 14 , they 'd become beggars and drug addicts . The rural poor had become the urban poor , and in the process they 'd become invisible . These heroin addicts in detox in Pakistan reminded me of figures in a play by Beckett : isolated , waiting in the dark , but drawn to the light . Agent Orange was a defoliant used during the Vietnam War to de
 ny cover to the Vietcong and the North Vietnamese army . The active ingredient was dioxin , an extremely toxic chemical that was sprayed in vast quantities , and whose effects passed through the genes to the next generation . In 2000 , I began documenting global health issues , concentrating first on AIDS in Africa . I tried to tell the story through the work of caregivers . I thought it was important to emphasize that people were being helped , whether by international NGOs or by local grassroots organizations . So many children have been orphaned by the epidemic that grandmothers have taken the place of parents , and a lot of children had been born with HIV . A hospital in Zambia . I begun documenting the close connection between HIV/AIDS and tuberculosis . This is an MSF hospital in Cambodia . My pictures can play a supporting role to the work of NGOs by shedding light on the critical social problems they 're trying to deal with . I went to Congo with MSF , and contributed to a b
 ook and an exhibition that focused attention on a forgotten war in which millions of people have died , and exposure to disease without treatment is used as a weapon . A malnourished child being measured as part of the supplemental feeding program . In the fall of 2004 I went to Darfur . This time I was on an assignment for a magazine , but again worked closely with MSF . The international community still has n't found a way to create the pressure necessary to stop this genocide . An MSF hospital in a camp for displaced people . I 've been working on a long project on crime and punishment in America . This is a scene from New Orleans . A prisoner on a chain gang in Alabama was punished by being handcuffed to a post in the midday sun . This experience raised a lot of questions , among them questions about race and equality and for whom in our country opportunities and options are available . In the yard of a chain gang in Alabama . I did n't see either of the planes hit . When I glan
 ced out my window , I saw the first tower burning , and I thought it might have been an accident . A few minutes later when I looked again and saw the second tower burning , I knew we were at war . In the midst of the wreckage at Ground Zero , I had a realization . I 'd been photographing in the Islamic world since 1981 -- not only in the Middle East , but also in Africa , Asia and Europe . At the time I was photographing in these different places , I thought I was covering separate stories . But on 9/11 history crystallized , and I understood I 'd actually been covering a single story for more than 20 years , and the attack on New York was its latest manifestation . The central commercial district of Kabul , Afghanistan at the end of the civil war , shortly before the city fell to the Taliban . Land mine victims being helped at the Red Cross rehab center being run by Alberto Cairo . A boy who lost a leg to a leftover mine . I 'd witnessed immense suffering in the Islamic world from
  political oppression , civil war , foreign invasions , poverty , famine . I understood that in its suffering , the Islamic world had been crying out . Why were n't we listening ? A Taliban fighter shot during a battle as the Northern Alliance entered the city of Kunduz . When war with Iraq was imminent , I realized the American troops would be very well covered , so I decided to cover the invasion from inside Baghdad . A marketplace was hit by a mortar shell that killed several members of a single family . A day after American forces entered Baghdad , a company of Marines began rounding up bank robbers and were cheered on by the crowds -- a hopeful moment that was short lived . For the first time in years , Shi'ites were allowed to make the pilgrimage to Karbala to observe Ashura , and I was amazed by the sheer number of people and how fervently they practiced their religion . A group of men march through the streets cutting themselves with knives . It was obvious that the Shi'ites
  were a force to be reckoned with , and we would do well to understand them and learn how to deal with them . Last year I spent several months documenting our wounded troops , from the battlefield in Iraq all the way home . This is a helicopter medic giving CPR to a soldier who had been shot in the head . Military medicine has become so efficient that the percentage of troops who survive after being wounded is much higher in this war than in any other war in our history . The signature weapon of the war is the IED , and the signature wound is severe leg damage . After enduring extreme pain and trauma , the wounded face a grueling physical and psychological struggle in rehab . The spirit they displayed was absolutely remarkable . I tried to imagine myself in their place , and I was totally humbled by their courage and determination in the face of such catastrophic loss . Good people had been put in a very bad situation for questionable results . One day in rehab someone , started tal
 king about surfing and all these guys who 'd never surfed before said , " Hey , lets go . " And they went surfing . Photographers go to the extreme edges of human experience to show people what 's going on . Sometimes they put their lives on the line , because they believe your opinions and your influence matter . They aim their pictures at your best instincts , generosity , a sense of right and wrong , the ability and the willingness to identify with others , the refusal to accept the unacceptable . My TED wish : there 's a vital story that needs to be told , and I wish for TED to help me gain access to it and then to help me come up with innovative and exciting ways to use news photography in the digital era . Thank you very much . ( Applause ) 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/103TeleHTC_Manual_0_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/103TeleHTC_Manual_0_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/103TeleHTC_Manual_0_EN.txt.txt
new file mode 100644
index 0000000..f314722
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/103TeleHTC_Manual_0_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeffUser Manual www . htc . com Please Read Before Proceeding THE BATTERY IS NOT CHARGED WHEN YOU TAKE IT OUT OF THE BOX . DO NOT REMOVE THE BATTERY PACK WHEN THE DEVICE IS CHARGING . YOUR WARRANTY IS INVALIDATED IF YOU DISASSEMBLE OR ATTEMPT TO DISASSEMBLE THE DEVICE . PRIVACY RESTRICTIONS Some countries require full disclosure of recorded telephone conversations , and stipulate that you must inform the person with whom you are speaking that the conversation is being recorded . Always obey the relevant laws and regulations of your country when using the recording feature of your PDA Phone . INTELLECTUAL PROPERTY RIGHT INFORMATION Copyright � 2009 HTC Corporation . All Rights Reserved . HTC , the HTC logo , HTC Innovation , ExtUSB , TouchFLO , HTC Touch Pro , Straight Talk , Push Internet and HTC Care are trademarks and/or service marks of HTC Corporation . Microsoft , Windows , Windows Mobile , Windows XP , Windows Vista , ActiveSync , Windows Mobile Device Center , Internet Explor
 er , MSN , Hotmail , Windows Live , Outlook , Excel , PowerPoint , Word , OneNote and Windows Media are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries . Bluetooth and the Bluetooth logo are trademarks owned by Bluetooth SIG , Inc. Wi-Fi is a registered trademark of the Wireless Fidelity Alliance , Inc. Java , J2ME and all other Java-based marks are trademarks or registered trademarks of Sun Microsystems , Inc. in the United States and other countries . Copyright � 2009 , Sun Microsystems , Inc. All Rights Reserved . Copyright � 2009 , Adobe Systems Incorporated . All Rights Reserved . microSD is a trademark of SD Card Association . Opera Mobile from Opera Software ASA . Copyright 1995-2009 Opera Software ASA . All Rights Reserved . Google , Google Maps , and YouTube are trademarks of Google , Inc. in the United States and other countries . Copyright � 2009 , Esmertec AG . All Rights Reserved . Copyright � 2003-2009
  , ArcSoft , Inc. and its licensors . All Rights Reserved . ArcSoft and the ArcSoft logo are registered trademarks of ArcSoft , Inc. in the United States and/or other countries . Copyright � 2004-2009 , Ilium Software , Inc. All Rights Reserved . Copyright � 2009 , Aplix Corporation . All Rights Reserved . Copyright � 2000 - 2008 WESTTEK LLC . All Rights Reserved . WESTTEK and the WT Logo are registered trademarks of WESTTEK , LLC . Licensed by QUALCOMM Incorporated under one or more of the following patents : 4,901,307 5,506,865 5,267,261 5,600,754 5,490,165 5,109,390 5,544,196 5,414,796 5,056,109 5,511,073 5,267,262 5,657,420 5,504,773 5,101,501 5,778,338 5,228,054 5,535,239 5,710,784 5,568,483 5,337,338 5,659,569 5,416,797 All other company , product and service names mentioned herein are trademarks , registered trademarks or service marks of their respective owners . HTC shall not be liable for technical or editorial errors or omissions contained herein , nor for incidental o
 r consequential damages resulting from furnishing this material . The information is provided \u201c as is \u201d without warranty of any kind and is subject to change without notice . HTC also reserves the right to revise the content of this document at any time without prior notice . No part of this document may be reproduced or transmitted in any form or by any means , electronic or mechanical , including photocopying , recording or storing in a retrieval system , or translated into any language in any form without prior written permission of HTC . Disclaimers THE WEATHER INFORMATION , STOCK DATA , DATA AND DOCUMENTATION ARE PROVIDED \u201c AS IS \u201d AND WITHOUT WARRANTY OR TECHNICAL SUPPORT OF ANY KIND FROM HTC . TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW , HTC AND ITS AFFILIATES expressly disclaim any and all representations and warranties , express or implied , arising by law or otherwise , regarding the Weather Information , Stock Data , Data , Documentation , or any other Prod
 ucts and services , including without limitation any express or implied warranty of merchantability , express or implied warranty of fitness for a particular purpose , non-infringement , quality , accuracy , completeness , effectiveness , reliability , usefulness , that the Weather Information , Stock Data , Data and/or Documentation will be error-free , or implied warranties arising from course of dealing or course of performance . Without limiting the foregoing , it is further understood that HTC and its Providers are not responsible for Your use or misuse of the Weather Information , Stock Data , Data and/or Documentation or the results from such use . HTC and its Providers make no express or implied warranties , guarantees or affirmations that weather information will occur or has occurred as the reports , forecasts , data , or information state , represent or depict and it shall have no responsibility or liability whatsoever to any person or entity , parties and non-parties ali
 ke , for any inconsistency , inaccuracy , or omission for weather or events predicted or depicted , reported , occurring or occurred . WITHOUT LIMITING THE GENERALITY OF THE FOREGOING , YOU ACKNOWLEDGE THAT THE WEATHER INFORMATION , STOCK DATA , DATA AND/ OR DOCUMENTATION MAY INCLUDE INACCURACIES AND YOU WILL USE COMMON SENSE AND FOLLOW STANDARD SAFETY PRECAUTIONS IN CONNECTION WITH THE USE OF THE WEATHER INFORMATION , STOCK DATA , DATA OR DOCUMENTATION . Limitation of Damages TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW , IN NO EVENT SHALL HTC OR ITS PROVIDERS BE LIABLE TO USER OR ANY THIRD PARTY FOR ANY INDIRECT , SPECIAL , CONSEQUENTIAL , INCIDENTAL OR PUNITIVE DAMAGES OF ANY KIND , IN CONTRACT OR TORT , INCLUDING , BUT NOT LIMITED TO , INJURY , LOSS OF REVENUE , LOSS OF GOODWILL , LOSS OF BUSINESS OPPORTUNITY , LOSS OF DATA , AND/OR LOSS OF PROFITS ARISING OUT OF , OR RELATED TO , IN ANY MANNER , OR THE DELIVERY , PERFORMANCE OR NONPERFORMANCE OF OBLIGATIONS , OR USE OF THE
  WEATHER INFORMATION , STOCK DATA , DATA OR DOCUMENTATION HEREUNDER REGARDLESS OF THE FORESEEABILITY THEREOF . Important Health Information and Safety Precautions When using this product , the safety precautions below must be taken to avoid possible legal liabilities and damages . Retain and follow all product safety and operating instructions . Observe all warnings in the operating instructions on the product . To reduce the risk of bodily injury , electric shock , fire , and damage to the equipment , observe the following precautions . ELECTRICAL SAFETY This product is intended for use when supplied with power from the designated battery or power supply unit . Other usage may be dangerous and will invalidate any approval given to this product . SAFETY PRECAUTIONS FOR PROPER GROUNDING INSTALLATION CAUTION : Connecting to an improperly grounded equipment can result in an electric shock to your device . This product is equipped with a USB Cable for connecting with desktop or notebook
  computer . Be sure your computer is properly grounded ( earthed ) before connecting this product to the computer . The power supply cord of a desktop or notebook computer has an equipment-grounding conductor and a grounding plug . The plug must be plugged into an appropriate outlet which is properly installed and grounded in accordance with all local codes and ordinances . SAFETY PRECAUTIONS FOR POWER SUPPLY UNIT \u2022 Use the correct external power source A product should be operated only from the type of power source indicated on the electrical ratings label . If you are not sure of the type of power source required , consult your authorized service provider or local power company . For a product that operates from battery power or other sources , refer to the operating instructions that are included with the product . \u2022 Handle battery packs carefully This product contains a Li-ion or Li-ion Polymer battery . There is a risk of fire and burns if the battery pack is handled improp
 erly . Do not attempt to open or service the battery pack . Do not disassemble , crush , puncture , short external contacts or circuits , dispose of in fire or water , or expose a battery pack to temperatures higher than 60�C ( 140�F ) . ARNING : Danger of explosion if battery is incorrectly replaced . To reduce risk W of fire or burns , do not disassemble , crush , puncture , short external contacts , expose to temperature above 60� C ( 140� F ) , or dispose of in fire or water . Replace only with specified batteries . Recycle or dispose of used batteries according to the local regulations or reference guide supplied with your product . \u2022 Take extra precautions \u2022 Keep the battery or device dry and away from water or any liquid as it may cause a short circuit . \u2022 Keep metal objects away so they don\u2019t come in contact with the battery or its connectors as it may lead to short circuit during operation . \u2022 The phone should only be connected to products that bear the USB-IF
  logo or have completed the USB-IF compliance program . \u2022 Do not use a battery that appears damaged , deformed , or discolored , or the one that has any rust on its casing , overheats , or emits a foul odor . \u2022 Always keep the battery out of the reach of babies and small children , to avoid swallowing of the battery . Consult the doctor immediately if the battery is swallowed . \u2022 If the battery leaks : \u2022 Do not allow the leaking fluid to come in contact with skin or clothing . If already in contact , flush the affected area immediately with clean water and seek medical advice . \u2022 Do not allow the leaking fluid to come in contact with eyes . If already in contact , DO NOT rub ; rinse with clean water immediately and seek medical advice . \u2022 Take extra precautions to keep a leaking battery away from fire as there is a danger of ignition or explosion . SAFETY PRECAUTIONS FOR DIRECT SUNLIGHT Keep this product away from excessive moisture and extreme temperatures . Do not leav
 e the product or its battery inside a vehicle or in places where the temperature may exceed 60�C ( 140�F ) , such as on a car dashboard , window sill , or behind a glass that is exposed to direct sunlight or strong ultraviolet light for extended periods of time . This may damage the product , overheat the battery , or pose a risk to the vehicle . PREVENTION OF HEARING LOSS CAUTION : Permanent hearing loss may occur if earphones or headphones are used at high volume for prolonged periods of time . NOTE : For France , Earphone ( listed below ) for this device have been tested to comply with the Sound Pressure Level requirement laid down in the applicable NF EN 503321:2000 and/or NF EN 50332-2:2003 standards as required by French Article L. 5232-1 . \u2022 Earphone , manufactured by HTC , Model HS S300 . SAFETY IN AIRCRAFT Due to the possible interference caused by this product to an aircraft\u2019s navigation system and its communications network , using this device\u2019s phone function on 
 board an airplane is against the law in most countries . If you want to use this device when on board an aircraft , remember to turn off your phone by switching to Airplane Mode . ENVIRONMENT RESTRICTIONS Do not use this product in gas stations , fuel depots , chemical plants or where blasting operations are in progress , or in potentially explosive atmospheres such as fuelling areas , fuel storehouses , below deck on boats , chemical plants , fuel or chemical transfer or storage facilities , and areas where the air contains chemicals or particles , such as grain , dust , or metal powders . Please be aware that sparks in such areas could cause an explosion or fire resulting in bodily injury or even death . EXPLOSIVE ATMOSPHERES When in any area with a potentially explosive atmosphere or where flammable materials exist , the product should be turned off and the user should obey all signs and instructions . Sparks in such areas could cause an explosion or fire resulting in bodily inju
 ry or even death . Users are advised not to use the equipment at refueling points such as service or gas stations , and are reminded of the need to observe restrictions on the use of radio equipment in fuel depots , chemical plants , or where blasting operations are in progress . Areas with a potentially explosive atmosphere are often , but not always , clearly marked . These include fueling areas , below deck on boats , fuel or chemical transfer or storage facilities , and areas where the air contains chemicals or particles , such as grain , dust , or metal powders . ROAD SAFETY Vehicle drivers in motion are not permitted to use telephony services with handheld devices , except in the case of emergency . In some countries , using handsfree devices as an alternative is allowed . SAFETY PRECAUTIONS FOR RF EXPOSURE \u2022 Avoid using your phone near metal structures ( for example , the steel frame of a building ) . \u2022 Avoid using your phone near strong electromagnetic sources , such as 
 microwave ovens , sound speakers , TV and radio . \u2022 Use only original manufacturer-approved accessories , or accessories that do not contain any metal . \u2022 Use of non-original manufacturer-approved accessories may violate your local RF exposure guidelines and should be avoided . INTERFERENCE WITH MEDICAL EQUIPMENT FUNCTIONS This product may cause medical equipment to malfunction . The use of this device is forbidden in most hospitals and medical clinics . If you use any other personal medical device , consult the manufacturer of your device to determine if they are adequately shielded from external RF energy . Your physician may be able to assist you in obtaining this information . Turn your phone OFF in health care facilities when any regulations posted in these areas instruct you to do so . Hospitals or health care facilities may be using equipment that could be sensitive to external RF energy . 10 HEARING AIDS Some digital wireless phones may interfere with some hearing aids .
  In the event of such interference , you may want to consult your service provider , or call the customer service line to discuss alternatives . NONIONIZING RADIATION Your device has an internal antenna . This product should be operated in its normaluse position to ensure the radiative performance and safety of the interference . As with other mobile radio transmitting equipment , users are advised that for satisfactory operation of the equipment and for the safety of personnel , it is recommended that no part of the human body be allowed to come too close to the antenna during operation of the equipment . Use only the supplied integral antenna . Use of unauthorized or modified antennas may impair call quality and damage the phone , causing loss of performance and SAR levels exceeding the recommended limits as well as result in non-compliance with local regulatory requirements in your country . To assure optimal phone performance and ensure human exposure to RF energy is within the 
 guidelines set forth in the relevant standards , always use your device only in its normal-use position . Contact with the antenna area may impair call quality and cause your device to operate at a higher power level than needed . Avoiding contact with the antenna area when the phone is IN USE optimizes the antenna performance and the battery life . Antenna location 11 General Precautions \u2022 Avoid applying excessive pressure to the device Do not apply excessive pressure on the screen and the device to prevent damaging them and remove the device from your pants \u2019 pocket before sitting down . It is also recommended that you store the device in a protective case and only use the device stylus or your finger when interacting with the touch screen . Cracked display screens due to improper handling are not covered by the warranty . \u2022 Device getting warm after prolonged use When using your device for prolonged periods of time , such as when you\u2019re talking on the phone , charging the
  battery or browsing the web , the device may become warm . In most cases , this condition is normal and therefore should not be interpreted as a problem with the device . \u2022 Heed service markings Except as explained elsewhere in the Operating or Service documentation , do not service any product yourself . Service needed on components inside the device should be done by an authorized service technician or provider . \u2022 Damage requiring service Unplug the product from the electrical outlet and refer servicing to an authorized service technician or provider under the following conditions : \u2022 Liquid has been spilled or an object has fallen onto the product . \u2022 The product has been exposed to rain or water . \u2022 The product has been dropped or damaged . \u2022 There are noticeable signs of overheating . \u2022 The product does not operate normally when you follow the operating instructions . \u2022 Avoid hot areas The product should be placed away from heat sources such as radiators , hea
 t registers , stoves , or other products ( including amplifiers ) that produce heat . 12 \u2022 Avoid wet areas Never use the product in a wet location . \u2022 Avoid using your device after a dramatic change in temperature When you move your device between environments with very different temperature and/or humidity ranges , condensation may form on or within the device . To avoid damaging the device , allow sufficient time for the moisture to evaporate before using the device . NOTICE : When taking the device from low-temperature conditions into a warmer environment or from high-temperature conditions into a cooler environment , allow the device to acclimate to room temperature before turning on power . \u2022 Avoid pushing objects into product Never push objects of any kind into cabinet slots or other openings in the product . Slots and openings are provided for ventilation . These openings must not be blocked or covered . \u2022 Mounting accessories Do not use the product on an unstable tab
 le , cart , stand , tripod , or bracket . Any mounting of the product should follow the manufacturer\u2019s instructions , and should use a mounting accessory recommended by the manufacturer . \u2022 Avoid unstable mounting Do not place the product with an unstable base . \u2022 Use product with approved equipment This product should be used only with personal computers and options identified as suitable for use with your equipment . \u2022 Adjust the volume Turn down the volume before using headphones or other audio devices . \u2022 Cleaning Unplug the product from the wall outlet before cleaning . Do not use liquid cleaners or aerosol cleaners . Use a damp cloth for cleaning , but NEVER use water to clean the LCD screen . 13 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/104TeleHTC_Manual_10_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/104TeleHTC_Manual_10_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/104TeleHTC_Manual_10_EN.txt.txt
new file mode 100644
index 0000000..715ed63
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Tele/104TeleHTC_Manual_10_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Chapter 10 Navigating on the Road 10.1 Google\u2122 Maps ( Available by Country ) Google\u2122 Maps lets you track your current location , view real-time traffic situations , and receive detailed directions to your destination . It also allows you to search and locate places of interest or even a contact address . Note You need an active data or Wi-Fi connection to use Google Maps . To open Google Maps , tap Start > Google Maps . Zoom in Zoom out View your current location . Get directions , see traffic information , and more . Search for places 190 Navigating on the Road Notes \u2022 Traffic information is available only in some selected cities . \u2022 Tap Menu > Help > Tips to learn more about the program and its many features . Searching a place of interest Google Maps lets you search and locate a place of interest , shows the contact information for that place , and gives directions on how to get there from your current location . 1. On the Google Maps screen , tap Search . 2. Enter the na
 me , address , or postal code of the place on the search field and then tap OK . 3. In the search results list , tap the place to view its address , phone number , and other details . Navigating on the Road 191 4. On the Address tab , tap : \u2022 The [ phone number ] to call the place of interest . \u2022 The web address ( if available ) to go to the web site of the place of interest . \u2022 Get directions to get directions to the place of interest from your current location . \u2022 Street view ( available by country ) to display a street view of the place of interest , when available . \u2022 Send as text message to send the place\u2019s contact information to your contact via SMS . \u2022 Save as favorite to add the place of interest to your list of favorite places . To access your favorite places , on the Google Maps screen , tap Menu > Favorites . \u2022 Save as contact to save the contact information of the place of interest to your contacts list . 5. Tap See map to view the place of interest on th
 e map . For more information about Google Maps , go to http://www.google.com/gmm/index.html. 10.2 Guidelines and Preparation for Using GPS If you installed a GPS navigation software on your device , follow these guidelines before and when using your device for GPS navigation . \u2022 When using GPS navigation software in the car , always plot your GPS route before starting to drive . Do not try to configure the GPS navigation software while driving . \u2022 The plotted GPS route is only for driving reference . It should not interfere with actual driving behavior . 192 Navigating on the Road \u2022 Do not leave your device in a vehicle or expose the device to direct sunlight to avoid overheating the battery , which could pose damage to the device or risk to the vehicle . \u2022 When using the device inside a vehicle , use the car holder to mount the device securely in place . Avoid placing the device in the areas shown in the following diagram : 1 Do not place where it will block the driver\u2019s 
 vision . 2 Do not place where air bags could deploy . 3 Do not place anywhere in the vehicle without securing the device in the holder . \u2022 Use the GPS system cautiously . Users shall be liable for any damages resulting from negligent operation of the system . \u2022 The GPS signal cannot pass through solid non-transparent objects . Signal reception may be affected by obstructions such as high-rise buildings , tunnels , bridges , forests , weather ( rainy or cloudy days ) , etc. If the vehicle\u2019s visor contains metal , it will be difficult for the GPS signal to pass through . \u2022 Other nearby wireless communication products ( such as mobile phones or radar-detecting devices ) may interfere with the satellite signal , resulting in unstable signal reception . Navigating on the Road 193 \u2022 The Global Positioning System ( GPS ) is built and operated by the US Defense Department . The Department is responsible for the accuracy and maintenance of the system . Any changes that the Departme
 nt makes may affect the accuracy and function of the GPS system . 10.3 Downloading Location Data via QuickGPS Before using the device for GPS navigation , open QuickGPS to download ephemeris data ( current satellite position and timing information ) . QuickGPS significantly speeds up the time for determining your GPS position . QuickGPS downloads ephemeris data from web servers , instead of from satellites , using your device\u2019s Internet connection via Wi-Fi , ActiveSync or GPRS/3G . To open QuickGPS Tap Start > All Programs > QuickGPS . To download GPS data Tap Download on the QuickGPS screen . On the screen , initially , you will see the Valid time of the downloaded data . As time passes , the remaining days and hours of validity will be shown . To speed up GPS positioning , download the latest ephemeris data when the validity of the data expires . 194 Navigating on the Road Changing the download options To configure download options , tap Menu > Options and choose from available
  options : \u2022 Remind me when data expires . Display a message when ephemeris data expires . \u2022 Auto download when data expires . Automatically download ephemeris data when it expires . \u2022 Auto download when connected to PC via ActiveSync . Automatically download ephemeris data when it is connected to your computer via ActiveSync . Your computer must be connected to the Internet to download ephemeris data . \u2022 Exit QuickGPS after downloading . Close QuickGPS after satellite data is downloaded . 
\ No newline at end of file


[18/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Unit/133UnitGass_Resolution_62250_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Unit/133UnitGass_Resolution_62250_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Unit/133UnitGass_Resolution_62250_EN.txt.txt
new file mode 100644
index 0000000..31a8f83
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Unit/133UnitGass_Resolution_62250_EN.txt.txt
@@ -0,0 +1,2 @@
+
+Resolution adopted by the General Assembly [ on the report of the Fifth Committee ( A/62/600/Add.1 ) ] Support account for peacekeeping operations The General Assembly , Recalling its resolutions 45/258 of 3 May 1991 , 47/218 A of 23 December 1992 , 48/226 A of 23 December 1993 , 55/238 of 23 December 2000 , 56/241 of 24 December 2001 , 56/293 of 27 June 2002 , 57/318 of 18 June 2003 , 58/298 of 18 June 2004 , 59/301 of 22 June 2005 , 60/268 of 30 June 2006 , 61/245 and 61/246 of 22 December 2006 , 61/256 of 15 March 2007 and 61/279 of 29 June 2007 , its decisions 48/489 of 8 July 1994 , 49/469 of 23 December 1994 and 50/473 of 23 December 1995 and its other relevant resolutions , Having considered the reports of the Secretary-General on the financing of the support account for peacekeeping operations and on the comprehensive analysis of the Office of Military Affairs of the Department of Peacekeeping Operations , the preliminary report of the Secretary-General on the status of impl
 ementation of General Assembly resolution 61/279 on strengthening the capacity of the United Nations to manage and sustain peacekeeping operations , the report of the Independent Audit Advisory Committee on the proposed budget for the Office of Internal Oversight Services under the support account for peacekeeping operations for the period from 1 July 2008 to 30 June 2009 and the related report of the Advisory Committee on Administrative and Budgetary Questions , Recognizing the importance of the ability of the United Nations to respond and deploy rapidly to a peacekeeping operation upon adoption of a relevant resolution of the Security Council , within thirty days for traditional peacekeeping operations and ninety days for complex peacekeeping operations , Recognizing also the need for adequate support during all phases of peacekeeping operations , including the liquidation and termination phases , Mindful that the level of the support account should broadly correspond to the manda
 te , number , size and complexity of peacekeeping missions , Takes note of the reports of the Secretary-General on the financing of the support account for peacekeeping operations1 and on the comprehensive analysis of the Office of Military Affairs of the Department of Peacekeeping Operations,2 the preliminary report of the Secretary-General on the status of implementation of General Assembly resolution 61/279 on strengthening the capacity of the United Nations to manage and sustain peacekeeping operations3 and the report of the Independent Audit Advisory Committee on the proposed budget for the Office of Internal Oversight Services under the support account for peacekeeping operations for the period from 1 July 2008 to 30 June 2009 ; 4 Reaffirms its role with regard to the structure of the Secretariat , and stresses that proposals that amend the overall departmental structure , as well as the format of the budgets of the Organization and the biennial programme plan , are subject to
  review and approval by the General Assembly ; Also reaffirms its role in carrying out a thorough analysis and approval of human and financial resources and policies with a view to ensuring the full , effective and efficient implementation of all mandated programmes and activities and the implementation of policies in this regard ; Further reaffirms that the Fifth Committee is the appropriate Main Committee of the General Assembly entrusted with responsibility for administrative and budgetary matters ; Reaffirms rule 153 of its rules of procedure ; Emphasizes that ongoing management reforms must be fully taken into account when presenting additional proposals for reform ; Reaffirms that the support account funds shall be used for the sole purpose of financing human resources and non-human resource requirements for backstopping and supporting peacekeeping operations at Headquarters , and that any changes in this limitation require the prior approval of the General Assembly ; Also rea
 ffirms the need for adequate funding for the backstopping of peacekeeping operations , as well as the need for justification for that funding in support account budget submissions ; Recalls the role of the Secretary-General as the Chief Administrative Officer of the Organization , in accordance with the provisions of Article 97 of the Charter of the United Nations ; Reiterates that the delegation of authority on the part of the Secretary-General should be in order to facilitate the better management of the Organization , but stresses that the overall responsibility for management of the Organization rests with the Secretary-General as the Chief Administrative Officer ; Affirms the need for the Secretary-General to ensure that the delegation of authority to the Department of Peacekeeping Operations , the Department of Field Support and field missions is in strict compliance with relevant resolutions and decisions , as well as relevant rules and procedures of the General Assembly on t
 his matter ; Stresses that heads of departments report to and are accountable to the Secretary-General ; Notes the unique nature of the reporting line from the head of the Department of Field Support to the Under-Secretary-General for Peacekeeping Operations , and decides that having one head of department ( Department of Field Support ) report to and take direction from another head of department ( Department of Peacekeeping Operations ) shall not set a precedent in the Secretariat ; Requests the Secretary-General to address systemic issues that hamper good management of the Organization , including by improving work processes and procedures , and in that context stresses that structural change is no substitute for managerial improvement ; Reiterates the importance of strengthened accountability in the Organization and of ensuring greater accountability of the Secretary-General to Member States , inter alia , for the effective and efficient implementation of legislative mandates an
 d the use of human and financial resources ; Recalls its request to the Secretary-General to specifically define accountability , as well as clear accountability mechanisms , including to the General Assembly , and to propose clear parameters for its application and the instruments for its rigorous enforcement , without exception , at all levels , to ensure effective and efficient operations and management of resources in the Organization ; Emphasizes the importance of preserving the unity of command in missions at all levels , as well as a coherence in policy and strategy and clear command structures in the field and up to and including Headquarters ; Requests the Secretary-General to ensure a clear chain of command , accountability , coordination and maintenance of an adequate system of checks and balances ; Emphasizes the importance of interaction and coordination with troop-contributing countries ; Also emphasizes the need to ensure the safety and security of United Nations pers
 onnel ; Urges the Secretary-General , within the framework established in its resolutions 52/12 B of 19 December 1997 and 52/220 of 22 December 1997 , to define explicitly the role and duties of the Deputy Secretary-General in the reform set out in its resolution 61/279 , including in relation to the Department of Peacekeeping Operations , the Department of Field Support , the Department of Political Affairs and the Department of Management ; Recalls section I , paragraph 6 , of its resolution 55/238 , paragraph 11 of its resolution 56/241 and paragraph 19 of its resolution 61/279 , and requests the Secretary-General to ensure the proper representation of troop-contributing countries in the Department of Peacekeeping Operations and the Department of Field Support , taking into account their contribution to United Nations peacekeeping ; Reiterates that the Secretary-General , in employing staff , shall ensure the highest standards of efficiency , competence and integrity as the param
 ount consideration , with due regard for the principle of equitable geographical distribution , in accordance with Article 101 , paragraph 3 , of the Charter and relevant resolutions of the General Assembly ; Reaffirms paragraph 67 of its resolution 61/279 , and requests the Secretary-General to include detailed information on mechanisms in place and measures taken to address the management challenges which the new organizational structure of the Department of Peacekeeping Operations/Department of Field Support poses , and the improvement that the new structure has brought in ensuring efficiency and effectiveness in the support for peacekeeping operations and special political missions , as well as to coordination with the Department of Political Affairs , in the context of the comprehensive report to be submitted at the second part of its resumed sixty-third session ; Notes with concern the late submission of the budgets of some peacekeeping operations , which puts considerable str
 ain on the work of the General Assembly and the Advisory Committee on Administrative and Budgetary Questions , and , while recognizing the challenges faced in preparing the budget proposals and related reports on peacekeeping and special factors affecting some missions , requests the Secretary-General to intensify his efforts to improve the quality and timely issuance of peacekeeping documents ; Reiterates its request in paragraph 13 of its resolution 60/268 and paragraph 32 of its resolution 61/279 , and urges the Secretary-General to submit the comprehensive report on the evolution of the support account at the second part of its resumed sixty-third session within the context of his next budget proposal for the support account ; Notes that the application of an accurate vacancy rate is good budgetary practice and essential for the appropriate assessment on Member States ; Requests the Secretary-General , when submitting his budget proposals , to include details of the full annual 
 cost of posts for the subsequent budget ; Takes note of paragraph 48 of the report of the Advisory Committee on Administrative and Budgetary Questions ; 5 Stresses the importance of complementarity of efforts and avoiding duplication between integrated operational teams and substantive components of the Secretariat , and requests the Secretary-General to report thereon , and to provide a clear definition of the roles and responsibilities of the integrated operational teams , in the comprehensive report to be submitted to the General Assembly at the second part of its resumed sixty-third session ; Affirms the importance of ensuring that the information and communications technology operations and requirements relating to peacekeeping are fully addressed and properly managed , taking into account the principle of unity of command ; Reaffirms the need for effective and efficient administration and financial management of peacekeeping operations , and urges the Secretary-General to cont
 inue to identify measures to increase the productivity and efficiency of the support account ; Requests the Secretary-General to ensure the full implementation of the relevant provisions of General Assembly resolutions 59/296 of 22 June 2005 , 60/266 of 30 June 2006 and 61/276 of 29 June 2007 and other relevant resolutions ; Endorses the conclusions and recommendations contained in the report of the Advisory Committee on Administrative and Budgetary Questions,5 subject to the provisions of the present resolution ; Takes note of paragraphs 81 to 87 of the report of the Advisory Committee on Administrative and Budgetary Questions,5 and decides to establish the following posts in the current structure of the Office of Military Affairs : ( a ) One D-1 , two P-5 , ten P-4 and four P-3 posts in the Office of the Military Adviser and one P-4 post for a civilian officer ; ( b ) Three P-4 posts and two P-3 posts in the Force Generation Service ; ( c ) Twelve P-4 posts in the Military Plannin
 g Service ; ( d ) Four P-4 posts in the Current Military Operations Service ; ( e ) One P-4 and three P-3 posts to be assigned to the Logistics Support Division of the Department of Field Support ; ( f ) One P-4 and one P-3 to be assigned to the Information and Communications Technology Division of the Department of Field Support ; Requests the Secretary-General to submit for its consideration at its sixty-fourth session a comprehensive report on the implementation of the strengthening of the Office of Military Affairs and its impact on the organization and capacities of the Office ; Requests the Special Committee on Peacekeeping Operations to consider the report referred to in paragraph 36 above at its 2010 substantive session ; Decides to approve the following posts : ( a ) One P-5 post for a Security Focal Point in the Situation Centre , Department of Peacekeeping Operations ; ( b ) Two P-4 posts for a Policy Adviser and a Policy Development Officer in the Police Division , Depar
 tment of Peacekeeping Operations ; ( c ) One P-3 post for a Programme Officer in the Risk Management Unit of the Office of the Under-Secretary-General for Field Support ; ( d ) One P-3 post for a Finance and Budget Officer in the Budget and Performance Reporting Service of the Field Budget and Finance Division , Department of Field Support ; ( e ) One D-2 post for the Director of the Information and Communications Technology Division , Department of Field Support ; ( f ) One General Service ( Other level ) post for a Human Resources Assistant ( Roster Development ) in the Office of Human Resources Management ; Decides not to approve the following posts : ( a ) One D-1 post for a Principal Officer and one General Service ( Other level ) post for an Administrative Assistant in the Asia and Middle East Division , Department of Peacekeeping Operations ; ( b ) One P-3 post in the Engineering Section of the Logistics Support Division , Department of Field Support ; ( c ) One P-4 post in t
 he Property Management Section of the Logistics Support Division , Department of Field Support ; ( d ) One P-4 post for a Technical Information Operations Support Officer , Department of Field Support ; ( e ) One P-4 post for a Management Analyst Officer in the Management Support Service of the Office of the Under-Secretary-General for Management ; ( f ) One General Service ( Other level ) Finance Assistant ( Health and Life Insurance Section ) in the Accounts Division , Department of Management ; ( g ) One P-4 and one P-3 post for Procurement Officers in the Procurement Division , Department of Management ; ( h ) One P-4 Legal Officer in the Office of the Legal Counsel , Office of Legal Affairs ; Decides to convert one P-5 and one General Service general temporary assistance position in the Conduct and Discipline Unit to posts ; Also decides to approve the following positions as general temporary assistance : ( a ) One P-4 Human Resources Officer ( Outreach and Strategic Staffing S
 ection ) in the Office of Human Resources Management , Department of Management ; ( b ) One P-3 Finance Officer in the Treasury Division , Department of Management ; Takes note of paragraph 130 of the report of the Advisory Committee on Administrative and Budgetary Questions,5 and decides to continue the four P-3 positions in the Peacekeeping Financing Division , Department of Management ; Decides to reduce the non-post resources by 1,899,100 United States dollars , and requests the Secretary-General to consider applying the reduction , inter alia , to the consultancy requirements referred to in paragraphs 297 and 354 of the report of the Secretary-General on the budget for the support account for peacekeeping operations for the period from 1 July 2008 to 30 June 2009 ; Also decides to maintain , for the financial period from 1 July 2008 to 30 June 2009 , the funding mechanism for the support account used in the current period , from 1 July 2007 to 30 June 2008 , as approved in para
 graph 3 of its resolution 50/221 B of 7 June 1996 ; Financial performance report for the period from 1 July 2006 to 30 June 2007 Takes note of the report of the Secretary-General on the financial performance of the support account for peacekeeping operations for the period from 1 July 2006 to 30 June 2007 ; Decides not to transfer the amount of 2,014,000 dollars included in the amount of 7,097,000 dollars , previously authorized in its resolution 61/279 , representing the excess of the authorized level of the Peacekeeping Reserve Fund utilized to finance the requirements of the support account in respect of the period from 1 July 2007 to 30 June 2008 ; Decides to apply the total amount of 13,790,000 dollars , comprising the unencumbered balance of 5,491,600 dollars and other income of 1,759,000 dollars in respect of the financial period ended 30 June 2007 , the support account fund balance in respect of financial periods ended 30 June 1997 , 30 June 1998 , 30 June 1999 and 30 June 2
 000 in the total amount of 2,138,000 dollars , and the excess of the authorized level of the Peacekeeping Reserve Fund in respect of the financial period ended 30 June 2007 in the amount of 4,401,400 dollars , to the support account requirements for the period from 1 July 2006 to 30 June 2007 ; Also decides to apply the excess of the authorized level of the Peacekeeping Reserve Fund in respect of the financial period ended 30 June 2007 in the amount of 2,014,000 dollars to the support account requirements for the period from 1 July 2007 to 30 June 2008 ; Budget estimates for the financial period from 1 July 2008 to 30 June 2009 Approves the support account requirements in the amount of 273,922,800 dollars for the financial period from 1 July 2008 to 30 June 2009 , including 1,122 continuing and 98 new temporary posts and their related post and non-post requirements ; Financing of the budget estimates Decides that the requirements for the support account for peacekeeping operations f
 or the financial period from 1 July 2008 to 30 June 2009 shall be financed as follows : ( a ) The amount of 469,600 dollars , representing the balance remaining of the excess of the authorized level of the Peacekeeping Reserve Fund in respect of the financial period ended 30 June 2007 , to be applied to the resources required for the financial period from 1 July 2008 to 30 June 2009 ; ( b ) The balance of 273,453,200 dollars to be prorated among the budgets of the active peacekeeping operations for the financial period from 1 July 2008 to 30 June 2009 ; ( c ) The estimated staff assessment income of 26,274,600 dollars , comprising the amount of 26,221,200 dollars for the financial period from 1 July 2008 to 30 June 2009 and the increase of 53,400 dollars in respect of the financial period ended 30 June 2007 , to be offset against the balance referred to in subparagraph ( b ) above , to be prorated among the budgets of the individual active peacekeeping operations . 109th plenary mee
 ting 20 June 2008 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Unit/134UnitUnat_Charter_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Unit/134UnitUnat_Charter_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Unit/134UnitUnat_Charter_EN.txt.txt
new file mode 100644
index 0000000..a2cbff9
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Unit/134UnitUnat_Charter_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Charter of the United Nations -------------------------------------------------------------------------------- INTRODUCTORY NOTE The Charter of the United Nations was signed on 26 June 1945 , in San Francisco , at the conclusion of the United Nations Conference on International Organisation , and came into force on 24 October 1945. The Statute of the International Court of Justice is an integral part of the Charter . Amendments to Articles 23 , 27 and 61 of the Charter were adopted by the General Assembly on 17 December 1963 and came into force on 31 August 1965. A further amendment to Article 61 was adopted by the General Assembly on 20 December 1971 , and came into force on 24 September 1973. An amendment to Article 109 , adopted by the General Assembly on 20 December 1965 , came into force on 12 June 1968. The amended Article 27 provides that decisions of the Security Council on procedural matters shall be made by an affirmative vote of nine members ( formerly seven ) and on 
 all other matters by an affirmative vote of nine members ( formerly seven ) , including the concurring votes of the five permanent members of the Security Council . The amendment to Article 61 , which entered into force on 31 August 1965 , enlarged the membership of the Economic and Social Council from eighteen to twenty-seven . The subsequent amendment to that Article , which entered into force on 24 September 1973 , further increased the membership of the Council from twenty-seven to fifty-four . The amendment to Article 109 , which relates to the first paragraph of that Article , provides that a General Conference of Member States for the purpose of reviewing the Charter may be held at a date and place to be fixed by a two-thirds vote of the members of the General Assembly and by a vote of any nine members ( formerly seven ) of the Security Council . Paragraph 3 of Article 109 , which deals with the consideration of a possible review conference during the tenth regular session of
  the General Assembly , has been retained in its original form in its reference to a \u201c vote , of any seven members of the Security Council \u201d , the paragraph having been acted upon in 1955 by the General Assembly , at its tenth regular session , and by the Security Council . PREAMBLE TO THE CHARTER OF THE UNITED NATIONS WE THE PEOPLES OF THE UNITED NATIONS , DETERMINED to save succeeding generations from the scourge of war , which twice in our lifetime has brought untold sorrow to mankind , and to reaffirm faith in fundamental human rights , in the dignity and worth of the human person , in the equal rights of men and women and of nations large and small , and to establish conditions under which justice and respect for the obligations arising from treaties and other sources of international law can be maintained , and to promote social progress and better standards of life in larger freedom , AND FOR THESE ENDS to practice tolerance and live together in peace with one another as 
 good neighbours , and to unite our strength to maintain international peace and security , and to ensure , by the acceptance of principles and the institution of methods , that armed force shall not be used , save in the common interest , and to employ international machinery for the promotion of the economic and social advancement of all peoples , HAVE RESOLED TO COMBINE OUR EFFORTS TO ACCOMPLISH THESE AIMS Accordingly , our respective Governments , through representatives assembled in the city of San Francisco , who have exhibited their full powers found to be in good and due form , have agreed to the present Charter of the United Nations and do hereby establish an international organization to be known as the United Nations . CHAPTER I PURPOSES AND PRINCIPLES Article 1 The Purposes of the United Nations are : 1. To maintain international peace and security , and to that end : to take effective collective measures for the prevention and removal of threats to the peace , and for th
 e suppression of acts of aggression or other breaches of the peace , and to bring about by peaceful means , and in conformity with the principles of justice and international law , adjustment or settlement of international disputes or situations which might lead to a breach of the peace ; 2. To develop friendly relations among nations based on respect for the principle of equal rights and self-determination of peoples , and to take other appropriate measures to strengthen universal peace ; 3. To achieve international co-operation in solving international problems of an economic , social , cultural , or humanitarian character , and in promoting and encouraging respect for human rights and for fundamental freedoms for all without distinction as to race , sex , language , or religion ; and 4. To be a centre for harmonizing the actions of nations in the attainment of these common ends . Article 2 The Organization and its Members , in pursuit of the Purposes stated in Article 1 , shall a
 ct in accordance with the following Principles . 1. The Organization is based on the principle of the sovereign equality of all its Members . 2. All Members , in order to ensure to all of them the rights and benefits resulting from membership , shall fulfil in good faith the obligations assumed by them in accordance with the present Charter . 3. All Members shall settle their international disputes by peaceful means in such a manner that international peace and security , and . justice , are not endangered . 4. All Members shall refrain in their international relations from the threat or use of force against the territorial integrity or political independence of any state , or in any other manner inconsistent with the Purposes of the United Nations . 5. All Members shall give the United Nations every assistance in any action it takes in accordance with the present Charter , and shall refrain from giving assistance to any state against which the United Nations is taking preventive or
  enforcement action . 6. The Organization shall ensure that states which are not Members of the United Nations act in accordance with these Principles so far as may be necessary for the maintenance of international peace and security . 7. Nothing contained in the present Charter shall authorize the United Nations to intervene in matters which are essentially within the domestic jurisdiction of any state or shall require the Members to submit such matters to settlement under the present Charter ; but this principle shall not prejudice the application of enforcement measures under Chapter VII . CHAPTER II MEMBERSHIP Article 3 The original Members of the United Nations shall be the states which , having participated in the United Nations Conference on International Organization at San Francisco , or having previously signed the Declaration by United Nations of 1 January 1942 , sign the present Charter and ratify it in accordance with Article 110. Article 4 1. Membership in the United N
 ations is open to a other peace-loving states which accept the obligations contained in the present Charter and , in the judgment of the Organization , are able and willing to carry out these obligations . 2. The admission of any such state to membership in the Nations will be effected by a decision of the General Assembly upon the recommendation of the Security Council . Article 5 A Member of the United Nations against which preventive or enforcement action has been taken by the Security Council may be suspended from the exercise of the rights and privileges of membership by the General Assembly upon the recommendation of the Security Council . The exercise of these rights and privileges may be restored by the Security Council . Article 6 A Member of the United Nations which has persistently violated the Principles contained in the present Charter may be ' expelled from the Organization by the General Assembly upon the recommendation of the Security Council . CHAPTER III ORGANS Art
 icle 7 1. There are established as the principal organs of the United Nations : a General Assembly , a Security Council , an Economic and Social Council , a Trusteeship Council , an International Court of Justice , and a Secretariat . 2. Such subsidiary organs as may be found necessary may be established in accordance with the present Charter . Article 8 The United Nations shall place no restrictions on the eligibility of men and women to participate in any capacity and under conditions of equality in its principal and subsidiary organs . CHAPTER IV THE GENERAL ASSEMBLY Composition Article 9 1. The General Assembly shall consist of all the Members of the United Nations . 2. Each Member shall have not more than five representatives in the General Assembly . Functions and Powers Article 10 The General Assembly may discuss any questions or any matters within the scope of the present Charter or relating to the powers and functions of any organs provided for in the present Charter , and 
 , except as provided in Article 12 , may make recommendations to the Members of the United Nations or to the Security Council or to both on any such questions or matters . Article 11 1. The General Assembly may consider the general principles of co-operation in the maintenance of international peace and security , including the principles governing disarmament and the regulation of armaments , and may make recommendations with regard to such principles to the Members or to the Security Council or to both . 2. The General Assembly may discuss any questions relating to the maintenance of international peace and security brought before it by any Member of the United Nations , or by the Security Council , or by a state which is not a Member of the United Nations in accordance with Article 35 , paragraph 2 , and , except as provided in Article 12 , may make recommendations with regard to any such questions to the state or states concerned or to the Security Council or to both . Any such 
 question on which action is necessary shall be referred to the Security Council by the General Assembly either before or after discussion . 3. The General Assembly may call the attention of the Security Council to situations which are likely to endanger international peace and security . 4. The powers of the General Assembly set forth in this Article shall not limit the general scope of Article 10. Article 12 1. While the Security Council is exercising in respect of any dispute or situation the functions assigned to it in the present Charter , the General Assembly shall not make any recommendation with regard to that dispute or situation unless the Security Council so requests . 2. The Secretary-General , with the consent of the Security Council , shall notify the General Assembly at each session of any matters relative to the maintenance of international peace and security which are being dealt with by the Security Council and similarly notify the General Assembly , or the Members 
 of the United Nations if the General Assembly is not in session , immediately the Security Council ceases to deal with such matters . Article 13 1. The General Assembly shall initiate studies and make recommendations for the purpose of : a. promoting international co-operation in the political field and encouraging the progressive development of international law and its codification ; b. promoting international co-operation in the economic , social , cultural , educational , and health fields , an assisting in the realization of human rights and fundamental freedoms for all without distinction as to race , sex , language , or religion . 2. The further responsibilities , functions and powers of the General with respect to matters mentioned in paragraph ) above are set forth in Chapters IX and X. Article 14 Subject to the provisions of Article 12 , the General Assembly may recommend measures for the peaceful adjustment of any situation , regardless of origin , which it deems likely t
 o impair the general welfare or friendly relations among nations , including situations resulting from a violation of the provisions of the present Charter setting forth the Purposes and Principles of the United Nations . Article 15 1. The General Assembly shall receive and consider annual and special reports from the Security Council ; these reports shall include an account of the measures that the Security Council has decided upon or taken to maintain international peace and security . 2. The General Assembly shall receive and consider reports from the other organs of the United Nations . Article 16 The General Assembly shall perform such functions with respect to the international trusteeship system as are assigned to it under Chapters XII and XIII , including the approval of the trusteeship agreements for areas not designated as strategic . Article 17 1. The Genera Assembly shall consider and approve the budget of the Organization . 2. The expenses of the Organization shall be b
 orne by the Members as apportioned by the General Assembly . 3. The Assembly shall consider and approve any financial and budgetary arrangements with specialize agencies referred to in Article 57 and shall examine the administrative budgets of such specialized agencies with a view to making recommendations to the agencies concerned . Voting Article 18 1. Each member of the General Assembly shall have one vote . 2. Decisions of the General Assembly on important questions shall be made by a two-thirds majority of the members present and voting . These questions shall include : recommendations with respect to the maintenance of international peace and security , the election of the non-permanent members of the Security Council , the election of the members of the Economic and Social Council , the election of members of the Trusteeship Council in accordance with paragraph 1 of Article 86 , the admission of new Members to the United Nations , the suspension of the rights and privileges o
 f membership , the expulsion of Members , questions relating to the operation of the trusteeship system , and budgetary questions . 3. Decisions on other questions , including the determination of additional categories of questions to be decided by a two-thirds majority , shall be made by a majority of the members present and voting . Article 19 A Member of the United Nations which is in arrears in the payment of its financial contributions to the Organization shall have no vote in the General Assembly if the amount of its arrears equals or exceeds the amount of the contributions due from it for the preceding two full years . The General Assembly may , nevertheless , permit such a Member to vote if it is satisfied that the failure to pay is due to conditions beyond the of the Member . Procedure Article 20 The General Assembly shall meet in regular annual sessions and in such special sessions as occasion may require . Special sessions shall be convoked by the Secretary-General at the
  request of the Security Council or of a majority of the Members of the United Nations . Article 21 The General Assembly shall adopt its own rules of procedure . It shall elect its President for each session . Article 22 The General Assembly may establish such subsidiary organs as it deems necessary for the performance of its functions . CHAPTER V THE SECURITY COUNCIL Composition Article 23 1. The Security Council shall consist of fifteen Members of the United Nations . The Republic of China , France , the Union of Soviet Socialist , the United Kingdom of Great Britain and Northern Ireland , and the United States of America shall be permanent members of the Security Council . The General Assembly shall elect ten other Members of the United Nations to be non-permanent members of the Security Council , due regard being specially paid , in the first instance to the contribution of Members of the United Nations to the maintenance of international peace and security and to the other purp
 oses of the Organization , and also to equitable geographical distribution . 2. The non-permanent members of the Security Council shall be elected for a term of two years . In the first election of the non-permanent members after the increase of the membership of the Security Council from eleven to fifteen , two of the four additional members shall be chosen for a term of one year . A retiring member shall not be eligible for immediate re-election . 3. Each member of the Security Council shall have one representative . Functions and Powers Article 24 1. In order to ensure prompt and effective action by the United Nations , its Members confer on the Security Council primary responsibility for the maintenance of international peace and security , and agree that in carrying out its duties under this responsibility the Security Council acts on their behalf . 2. In discharging these duties the Security Council shall act in accordance with the Purposes and Principles of the United Nations
  . The specific powers granted to the Security Council for the discharge of these duties are laid down in Chapters VI , VII , VIII , and XII . 3. The Security Council shall submit annual and , when necessary , special reports to the General Assembly for its consideration . Article 25 The Members of the United Nations agree to accept and carry out the decisions of the Security Council in accordance with the present Charter . Article 26 In order to promote the establishment and maintenance of international peace and security with the least diversion for armaments of the world 's human and economic resources , the Security Council shall be responsible for formulating , with the assistance of the Military Staff Committee referred to in Article 47 , plans to be submitted to the Members of the United-Nations for the establishment of a system for the regulation of armaments . Voting Article 27 1. Each member of the Security Council shall have one vote . 2. Decisions of the Security Council
  on procedural matters shall be made by an affirmative vote of nine members . 3. Decisions of the Security Council on all other matters shall be made by an affirmative vote of nine members including the concurring votes of the permanent members ; provided that , in decisions under Chapter VI , and under paragraph 3 of Article 52 , a party to a dispute shall abstain from voting . Procedure Article 28 1. The Security Council shall be so organized as to be able to function continuously . Each member of the Security Council shall for this purpose be represented at times at the seat of the Organization . 2. The Security Council shall hold meetings at which each of its members may , if it so desires , be represented by a member of the government or by some other specially designated representative . 3. The Security Council may hold meetings at such places other than the seat of the Organization as in its judgment will best facilitate its work . Article 29 The Security Council may establis
 h such subsidiary organs as it deems necessary for the performance of its functions . Article 30 The Security Council shall adopt its own rules of procedure , including the method of selecting its President . Article 31 Any Member of the United Nations which is not a member of the Security Council may participate , without vote , in the discussion of any question brought before the Security Council whenever the latter considers that the interests of that Member are specially affected . Article 32 Any Member of the United Nations which is not a member of the Security Council or any state which is not a Member of the United Nations , if it is a party to a dispute under consideration by the Security Council , shall be invited to participate , without vote , in the discussion relating to the dispute . The Security Council shall any down such conditions as it deems just for the participation of a state which is not a Member of the United Nations . CHAPTER VI PACIFIC SETTLEMENT OF DISPUTE
 S Article 33 1. The parties to any dispute , the continuance of which is likely to endanger the maintenance of international peace and security , shall , first of a , seek a solution by negotiation , enquiry , mediation , conciliation , arbitration , judicial settlement , resort to regional agencies or arrangements , or other peaceful means of their own choice . 2. The Security Council shall , when it deems necessary , call upon the parties to settle their dispute by such means . Article 34 The Security Council may investigate any dispute , or any situation which might lead to international friction or give rise to a dispute , in order to determine whether the continuance of the dispute or situation is likely to endanger the maintenance of international peace and security . Article 35 l. Any Member of the United Nations may bring any dispute , ~r any situation of the nature referred to in Article 34 , to the attention of the Security Council or of the General Assembly . 2. A state w
 hich is not a Member of the United Nations may bring to the attention of the Security Council or of the General Assembly any dispute to which it is a party if it accepts in advance , for the purposes of the dispute , the obligations of pacific settlement provided in the present Charter . 3. The proceedings of the General Assembly in respect of matters brought to its attention under this Article will be subject to the provisions of Articles 11 and 12. Article 36 1. The Security Council may , at any stage of a dispute of the nature referred to in Article 33 or of a situation of like nature , recommend appropriate procedures or methods of adjustment . 2. The Security Council should take into consideration any procedures for the settlement of the dispute which have already been adopted by the parties . 3. In making recommendations under this Article the Security Council should also take into consideration that legal disputes should as a general rule be referred by the parties to the Int
 ernational Court of Justice in accordance with the provisions of the Statute of the Court . Article 37 1. Should the parties to a dispute of the nature referred to in Article 33 fail to settle it by the means indicated in that Article , they shall refer it to the Security Council . 2. If the Security Council deems that the continuance of the dispute is in fact likely to endanger the maintenance of international peace and security , it shall decide whether to take action under Article 36 or to recommend such terms of settlement as it may consider appropriate . Article 38 Without prejudice to the provisions of Articles 33 to 37 , the Security Council may , if all the parties to any dispute so request , make recommendations to the parties with a view to a pacific settlement of the dispute . CHAPTER VII ACTION WITH RESPECT TO THREATS TO THE PEACE , BREACHES OF THE PEACE , AND ACTS OF AGGRESSION Article 39 The Security Council shall determine the existence of any threat to the peace , br
 each of the peace , or act of aggression and shall make recommendations , or decide what measures shall be taken in accordance with Articles 4 and 42 , to maintain or restore international peace and security . Article 40 In order to prevent an aggravation of the situation , the Security Council may , before making the recommendations or deciding upon the measures provided for in Article 39 , call upon the parties concerned to comply with such provisional measures as it deems necessary or desirable . Such provisional measures shall be without prejudice to the rights , claims , or position of the parties concerned . The Security Council shall duly take account of failure to comply with such provisional measures . Article 41 The Security Council may decide what measures not involving the use of armed force are to be employed to give effect to its decisions , and it may call upon the Members of the United Nations to apply such measures . These may include complete or partial interruptio
 n of economic relations and of rail , sea , air , postal , telegraphic , radio , and other means of communication , and the severance of diplomatic relations . Article 42 Should the Security Council consider that measures provided for in Article 41 would be inadequate or have proved to be inadequate , it may take such action by air , sea , or land forces as may be necessary to maintain or restore international peace and security . Such action may include demonstrations , blockade , and other operations by air , sea , or land forces of Members of the United Nations . Article 43 1. All Members of the United Nations , in order to contribute to the maintenance of international peace and security , undertake to make available to the Security Council , on its and in accordance with a special agreement or agreements , armed forces , assistance , and facilities , including rights of passage , necessary for the purpose of maintaining international peace and security . 2. Such agreement or ag
 reements shall govern the numbers and types of forces , their degree of readiness and general location , and the nature of the facilities and assistance to be provided . 3. The agreement or agreements shall be negotiated as soon as possible on the initiative of the Security Council . They shall be concluded between the Security Council and Members or between the Security Council and groups of Members and shall be subject to ratification by the signatory states in accordance with their respective constitutional processes . Article 44 When Security Council has decided to use force it shall , before calling upon a Member not represented on it to provide armed forces in fulfilment of the obligations assumed under Article 43 , invite that Member , if the Member so desires , to participate in the decisions of the Security Council concerning the employment of contingents of that Member 's armed forces . Article 45 In order to enable the Nations to take urgent military measures , Members sh
 all hold immediately available national air-force contingents for combined international enforcement action . The strength and degree of readiness of these contingents and plans for their combined action shall be determined , within the limits laid down in the special agreement or agreements referred to in Article 43 , by the Security Council with the assistance of the Military Committee . Article 46 Plans for the application of armed force shall be made by the Security Council with the assistance of the Military Staff Committee . Article 47 1. There shall be established a Military Staff Committee to advise and assist the Security Council on questions relating to the Security Council 's military requirements for the maintenance of international peace and security , the employment and command of forces placed at its disposal , the regulation of armaments , and possible disarmament . 2. The Military Staff Committee consist of the Chiefs of Staff of the permanent members of the Securit
 y Council or their representatives . Any Member of the United Nations not permanently represented on the Committee shall be invited by the Committee to be associated with it when the efficient discharge of the Committee 's responsibilities requires the participation of that Member its work . 3. The Military Staff Committee be responsible under the Security Council for the strategic direction of any armed forces paced at the disposal of the Security Council . Questions relating to the command of such forces shall be worked out subsequently . 4. The Military Staff Committee , with the authorization of the security Council and after consultation with appropriate regional agencies , may establish sub-committees . Article 48 1. The action required to carry out the decisions of the Security Council for the maintenance of international peace and security shall be taken by all the Members of the United Nations or by some of them , as the Security Council may determine . 2. Such decisions sh
 all be carried out by the Members of the United Nations directly and through their action in the appropriate international agencies of which they are members . Article 49 The Members of the United Nations shall join in affording mutual assistance in carrying out the measures decided upon by the Security Council . Article 50 If preventive or enforcement measures against any state are taken by the Security Council , any other state , whether a Member of the United Nations or not , which finds itself confronted with special economic problems arising from the carrying out of those measures shall have the right to consult the Security Council with regard to a solution of those problems . Article 51 Nothing in the present Charter shall impair the inherent right of individual or collective self-defence if an armed attack occurs against a Member of the United Nations , until the Security Council has taken measures necessary to maintain international peace and security . Measures taken by Me
 mbers in the exercise of this right of self-defence shall be immediately reported to the Security Council and shall not in any way affect the authority and responsibility of the Security Council under the present Charter to take at any time such action as it deems necessary in order to maintain or restore international peace and security . CHAPTER VIII REGIONAL ARRANGEMENTS Article 52 1. Nothing in the present Charter the existence of regional arrangements or agencies for dealing with such matters relating to the maintenance of international peace and security as are appropriate fur regional action , provided that such arrangements or agencies and their activities are consistent with the Purposes and Principles of the United Nations . 2. The Members of the United Nations entering into such arrangements or constituting such agencies shall make every effort to achieve pacific settlement of local disputes through such regional arrangements or by such regional agencies before referring 
 them to the Security Council . 3. The Security Council shall encourage the development of pacific settlement of local disputes through such regional arrangements or by such regional agencies either on the initiative of the states concerned or by reference from the Security Council . 4. This Article in no way the application of Articles 34 and 35. Article 53 1. The Security Council shall , where appropriate , utilize such regional arrangements or agencies for enforcement action under its authority . But no enforcement action shall be taken under regional arrangements or by regional agencies without the authorization of the Security Council , with the exception of measures against any enemy state , as defined in paragraph 2 of this Article , provided for pursuant to Article 107 or in regional arrangements directed against renewal of aggressive policy on the part of any such state , until such time as the Organization may , on request of the Governments concerned , be charged with the 
 responsibility for preventing further aggression by such a state . 2. The term enemy state as used in paragraph 1 of this Article applies to any state which during the Second World War has been an enemy of any signatory of the present Charter . Article 54 The Security Council shall at all times be kept fully informed of activities undertaken or in contemplation under regional arrangements or by regional agencies for the maintenance of international peace and security . CHAPTER IX INTERNATIONAL ECONOMIC AND SOCIAL CO-OPERATION Article 55 With a view to the creation of conditions of stability and well-being which are necessary for peaceful and friendly relations among nations based on respect for the principle of equal rights and self-determination of peoples , the United Nations shall promote : a. higher standards of living , fu employment , and conditions of economic and social progress and development ; b. solutions of international economic , social , health , and related problems
  ; and international cultural and educational cooperation ; and c. universal respect for , and observance of , human rights and fundamental freedoms for all without distinction as to race , sex , language , or religion . Article 56 All Members pledge themselves to take joint and separate action in co-operation with the Organization for the achievement of the purposes set forth in Article 55. Article 57 1. The various specialized agencies , established by intergovernmental agreement and having wide international responsibilities , as defined in their basic instruments , in economic , social , cultural , educational , health , and related fields , shall be brought into relationship with the United Nations in accordance with the provisions of Article 63. 2. Such agencies thus brought into relationship with the United Nations are hereinafter referred to as specialized agencies . Article 58 The Organization shall make recommendations for the co-ordination of the policies and activities o
 f the specialized agencies . Article 59 The Organization shall , where appropriate , initiate negotiations among the states concerned for the creation of any new specialized agencies required for the accomplishment of the purposes set forth in Article 55. Article 60 Responsibility for the discharge of the functions of the Organization set forth in this Chapter shall be vested in the General Assembly and , under the authority of the General Assembly , in the Economic and Social Council , which shall have for this purpose the powers set forth in Chapter X. CHAPTER X THE ECONOMIC AND SOCIAL COUNCIL Composition Article 61 1. The Economic and Social Council shall consist of fifty-four Members of the United Nations elected by the General Assembly . 2. Subject to the provisions of paragraph 3 , eighteen members of the Economic and Social Council shall be elected each year for a term of three years . A retiring member shall be eligible for immediate re-election . 3. At the first election af
 ter the increase in the membership of the Economic and Social Council from twenty-seven to fifty-four members , in addition to the members elected in place of the nine members whose term of office expires at the end of that year , twenty-seven additional members shall be elected . Of these twenty-seven additional members , the term of office of nine members so elected shall expire at the end of one year , and of nine other members at the end of two years , in accordance with arrangements made by the General Assembly . 4. Each member of the Economic and Social Council shall have one representative . Functions and Powers Article 62 1. The Economic and Social Council may make or initiate studies and reports with respect to international economic , social , cultural , educational , health , and related matters and may make recommendations with respect to any such matters to the General Assembly , to the Members of the United Nations , and to the specialized agencies concerned . 2. It ma
 y make recommendations for the purpose of promoting respect for , and observance of , human rights and fundamental freedoms for all . 3. It may prepare draft conventions for submission to the General Assembly , with respect to matters falling within its competence . 4. It may call , in accordance with the rules prescribed by the United Nations , international conferences on matters falling within its competence . Article 63 1. The Economic and Social Council may enter into agreements with any of the agencies referred to in Article 57 , defining the terms on which the agency concerned shall be brought into relationship with the United Nations . Such agreements shall be subject to approval by the General Assembly . 2. It may co-ordinate the activities of the specialized agencies through consultation with and recommendations to such agencies and through recommendations to the General Assembly and to the Members of the United Nations . Article 64 1. The Economic and Social Council may t
 ake appropriate steps to obtain regular reports from the specialized agencies . may make arrangements with the Members of the United Nations and with the specialized agencies to obtain reports on the steps taken to give effect to its own recommendations and to recommendations on matters falling within its competence made by the General Assembly . 2. It may communicate its observations on these reports to the General Assembly . Article 65 The Economic and Social Council may furnish information to the Security Council and shall assist the Security Council upon its request . Article 66 1. The Economic and Social Council shall perform such functions as fall within its competence in connexion with the carrying out of the recommendations of the General Assembly . 2. It may , with the approval of the General Assembly , perform services at the request of Members of the United Nations and at the request of specialized agencies . 3. It shall perform such other functions as are specified elsew
 here in the present Charter or as may be assigned to it by the General Assembly . Voting Article 67 1. Each member of the Economic and Social Council shall have one vote . 2. Decisions of the Economic and Social Council shall be made by a majority of the members present and voting . Procedure Article 68 The Economic and Social Council shall set up commissions in economic and social fields and for the promotion of human rights , and such other commissions as may for the performance of its functions . Article 69 The Economic and Social Council shall invite any Member of the United Nations to participate , without vote , in its deliberations on any matter of particular concern to that Member . Article 70 The Economic and Social Council may make arrangements for representatives of the specialized agencies to participate , without vote , in its deliberations and in those of the commissions established by it , and for its representatives to participate in the deliberations of the speciali
 zed agencies . Article 71 The Economic and Social Council may make suitable arrangements for consultation with non-governmental organizations which are concerned with matters within its competence . Such arrangements may be made with international organizations and , where appropriate , with national organizations after consultation with the Member of the United Nations concerned . Article 72 1. The Economic and Social Council shall adopt its own rules of procedure , including the method of selecting its President . 2. The Economic and Social Council shall meet as required in accordance with its rules , which shall include provision for the convening of meetings on the request of a majority of its members . CHAPTER XI DECLARATION REGARDING NON-SELF-GOVERNING TERRITORIES Article 73 Members of the United Nations which have or assume responsibilities for the administration of territories whose peoples have not yet attained a full measure of self-government recognize the principle that 
 the interests of the inhabitants of these territories are paramount , and accept as a sacred trust the obligation to promote to the utmost , within the system of international peace and security established by the present Charter , the well-being of the inhabitants of these territories , and , to this end : a. to ensure , with due respect for the culture of the peoples concerned , their political , economic , social , and educational advancement , their just treatment , and their protection against abuses ; b. to develop self-government , to take due account of the political aspirations of the peoples , and to assist them in the progressive development of their free political institutions , according to the particular circumstances of each territory and its peoples and their varying stages of advancement ; c. to further international peace and security ; d. to promote constructive measures of development , to encourage research , and to co-operate with one another and , when and whe
 re appropriate , with specialized international bodies with a view to the practical achievement of the social , economic , and scientific purposes set forth in this Article ; and e. to transmit regularly to the Secretary-General for information purposes , subject to such limitation as security and constitutional considerations may require , statistical and other information of a technical nature relating to economic , social , and educational conditions in the territories for which they are respectively responsible other than those territories to which Chapters XII and XIII apply . Article 74 Members of the United Nations also agree that their policy in respect of the territories to which this Chapter applies , no less than in respect of their metropolitan areas , must be based on the general principle of good-neigh-bourliness , due account being taken of the interests and well-being of the rest of the world , in social , economic , and commercial matters . CHAPTER XII INTERNATIONAL
  TRUSTEESHIP SYSTEM Article 75 The United Nations shall establish under its authority an international trusteeship system for the administration and supervision of such territories as may be placed thereunder by subsequent individual agreements . These territories are hereinafter referred to as trust territories . Article 76 The basic objectives of the trusteeship system , in accordance with the Purposes of the United Nations laid down in Article 1 of the present Charter , shall be : a. to further international peace and security ; b. to promote the political , economic , social , and educational advancement of the inhabitants of the trust territories , and their progressive development towards self-government or independence as may be appropriate to the particular circumstances of each territory and its peoples and the freely expressed wishes of the peoples concerned , and as may be provided by the terms of each trusteeship agreement ; c. to encourage respect for human rights and f
 or fundamental freedoms for all without distinction as to race , sex , language , or religion , and to encourage recognition of the interdependence of the peoples of the world ; and d. to ensure equal treatment in social , economic , and commercial matters for all Members of the United Nations and their , and also equal treatment for the latter in the administration of justice , without prejudice to the attainment of the foregoing objectives and subject to the provisions of Article 80. Article 77 1. The trusteeship system shall apply to such territories in the following categories as may be placed thereunder by means of trusteeship agreements : a. territories now held under mandate ; b. territories which may be detached from enemy states as a result of the Second World War ; and c. territories voluntarily placed under the system by states responsible for their administration . 2. It will be a matter for subsequent agreement as to which territories in the foregoing categories will be
  brought under the trusteeship system and upon what terms . Article 78 The trusteeship system shall not apply to territories which have become Members of the United Nations , relationship among which shall be based on respect for the principle of sovereign equality . Article 79 The terms of trusteeship for each territory to be placed under the trusteeship system , including any alteration or amendment , shall be agreed upon by the states directly concerned , including the mandatory power in the case of territories held under mandate by a Member of the United Nations , and shall be approved as provided for in Articles 83 and 85. Article 80 1. Except as may be agreed upon in individual trusteeship agreements , made under Articles 77 , 79 , and 81 , placing each territory under the trusteeship system , and until such agreements have been concluded , nothing in this Chapter shall be construed in or of itself to alter in any manner the rights whatsoever of any states or any peoples or th
 e terms of existing international instruments to which Members of the United Nations may respectively be parties . 2. Paragraph 1 of this Article shall not be interpreted as giving grounds for delay or postponement of the negotiation and conclusion of agreements for placing mandated and other territories under the trusteeship system as provided for in Article 77. Article 81 The trusteeship agreement shall in each case include the terms under which the trust territory will be administered and designate the authority which will exercise the administration of the trust territory . Such authority , hereinafter called the administering authority , may be one or more states or the Organization itself . Article 82 There may be designated , in any trusteeship agreement , a strategic area or areas which may include part or all of the trust territory to which the agreement applies , without prejudice to any special agreement or agreements made under Article 43. Article 83 1. All functions of 
 the United Nations relating to strategic areas , including the approval of the terms of the trusteeship agreements and of their alteration or amendment , shall be exercised by the Security Council . 2. he basic objectives set forth in Article 76 shall be applicable to the people of each strategic area . 43 3. The Security Council shall , subject to the provisions of the trusteeship agreements and without prejudice to security considerations , avail itself of the assistance of the Trusteeship Council to perform those functions of the United Nations under the trusteeship system relating to political , economic , social , and educational matters in the strategic areas . Article 84 It shall be the duty of the administering authority to ensure that the trust territory shall play its part in the maintenance of international peace and security . To this end the administering authority may make use of volunteer forces , facilities , and assistance from the trust territory in carrying out th
 e obligations towards the Security Council undertaken in this regard by the administering authority , as well as for local defence and the maintenance of law and order within the trust territory . Article 85 1. The functions of the United Nations with regard to trusteeship agreements for all areas not designated as strategic , including the approval of the terms of the trusteeship agreements and of their alteration or amendment , shall be exercised by the General Assembly . 2. The Trusteeship Council , operating under the authority of the General Assembly , shall assist the General Assembly in carrying out these functions . CHAPTER XIII THE TRUSTEESHIP COUNCIL Composition Article 86 1. The Trusteeship Council shall consist of the following Members of the United Nations : a. those Members administering trust territories ; b. such of those Members mentioned by name in Article 23 as are not administering trust territories ; and c. as many other Members elected for three-year terms by t
 he General Assembly as may be necessary to ensure that the total number of members of the Trusteeship Council is equally divided between those Members of the United Nations which administer trust territories and those which do not . 2. Each member of the Trusteeship Council shall designate one specially qualified person to represent it therein . Functions and Powers Article 87 The General Assembly and , under its authority , the Trusteeship Council , in carrying out their functions , may : a. consider reports submitted by the administering authority ; b. accept petitions and examine them in consultation with the administering authority ; c. provide for periodic visits to the respective trust territories at times agreed upon with the administering authority ; and d. take these and other actions in conformity with the terms of the trusteeship agreements . Article 88 The Trusteeship Council shall formulate a questionnaire on the political , economic , social , and educational advanceme
 nt of the inhabitants of each trust territory , and the administering authority for each trust territory within the competence of the General Assembly shall make an annual report to the General Assembly upon the basis of such questionnaire . Voting Article 89 1. Each member of the Trusteeship Council shall have one vote . 2. Decisions of the Trusteeship Council shall be made by a majority of the members present and voting . Procedure Article 90 1. The Trusteeship Council shall adopt its own rules of procedure , including the method of selecting its President . 2. The Trusteeship Council shall meet as required in accordance with its rules , which shall include provision for the convening of meetings on the request of a majority of its members . Article 91 The Trusteeship Council shall , when appropriate , avail itself of the assistance of the Economic and Social Council and of the specialized agencies in regard to matters with which they are respectively concerned . CHAPTER XIV THE I
 NTERNATIONAL COURT OF JUSTICE Article 92 The International Court of Justice shall be the principal judicial organ of the United Nations . It shall function in accordance with the annexed Statute , which is based upon the Statute of the Permanent Court of International Justice and forms an integral part of the present Charter . Article 93 1. All Members of the United Nations are facto parties to the Statute of the International Court of Justice . 2. A state which is not ~ of the United Nations may become a party to the Statute of the International Court of Justice on to be determined in each case by the General Assembly upon the recommendation of the Security Council . Article 94 1. Each Member of the United Nations undertakes to comply with the decision of the International Court of Justice in any case to which it is a party . 2. If any party to a case fails to perform the obligations incumbent upon it under a judgment rendered by the Court , the other party may have recourse to the
  Security Council , which may , if it deems necessary , make recommendations or decide upon measures to be taken to give to the judgment . Article 95 Nothing in the present Charter shall prevent Members of the United Nations from entrusting the solution of their differences to other tribunals by virtue of agreements already in existence or which may be concluded in the future . Article 96 1. The General Assembly or the Security Council may request the International Court of Justice to give an advisory opinion on any legal question . 2. Other organs of the United Nations and specialized agencies , which may at any time be so authorized by the General Assembly , may also request advisory opinions of the Court on legal questions arising within the scope of their activities . CHAPTER XV THE SECRETARIAT Article 97 The Secretariat shall comprise a Secretary-General and such staff as the Organization may require . The Secretary-General shall be appointed by the General Assembly upon the re
 commendation of the Security Council . He shall be the chief administrative officer of the Organization . Article 98 The Secretary-General shall act in that capacity in all meetings of the General Assembly , of the Security Council , of the Economic and Social Council , and of the Trusteeship Council , and shall perform such other functions as are entrusted to him by these organs . The Secretary-General shall make an annual report to the General Assembly on the work of the Organization . Article 99 The Secretary-General may bring to the attention of the Security Council any matter which in his opinion may threaten the maintenance of international peace and security . Article 100 1. In the performance of their duties the Secretary-General and the staff shall not seek or receive instructions from any government or from any other authority externa to the Organization . They shall refrain from any action which might on their position as international officials responsible only to the Or
 ganization . 2. Each Member of the United Nations undertakes to respect the exclusively international character of the responsibilities of the Secretary-General and the staff and not to seek to influence them in the discharge of their responsibilities . Article 101 1. The staff shall be appointed by the Secretary-General under regulations established by the General Assembly . 2. Appropriate staffs shall be permanently assigned to the Economic and Social Council , the Trusteeship Council , and , as required , to other organs of the United Nations . These staffs shall form a part of the Secretariat . 3. The paramount consideration in the employment of the staff and in the determination of the conditions of service shall be the necessity of securing the highest standards of efficiency , competence , and integrity . Due regard shall be paid to the importance of recruiting the staff on as wide a geographical basis as possible . CHAPTER XVI MISCELLANEOUS PROVISIONS Article 102 1. Every tr
 eaty and every international agreement entered into by any Member of the United Nations after the present Charter comes into force shall as soon as possible be registered with the Secretariat and published by it . 2. No party to any such treaty or international agreement which has not been registered in accordance with the provisions of paragraph I of this Article may invoke that treaty or agreement before any organ of the United Nations . Article 103 In the event of a conflict between the obligations of the Members of the United Nations under the present Charter and their obligations under any other international agreement , their obligations under the present Charter shall prevail . Article 104 The Organization shall enjoy in the territory of each of its Members such legal capacity as may be necessary for the exercise of its functions and the fulfilment of its purposes . Article 105 1. The Organization shall enjoy in the territory of each of its Members such privileges and immunit
 ies as are necessary for the fulfilment of its purposes . 2. Representatives of the Members of the United Nations and officials of the Organization shall similarly enjoy such privileges and immunities as are necessary for the independent exercise of their functions in connexion with the Organization . 3. The General Assembly may make recommendations with a view to determining the details of the application of paragraphs 1 and 2 of this Article or may propose conventions to the Members of the United Nations for this purpose . CHAPTER XVII TRANSITIONAL SECURITY ARRANGEMENTS Article 106 Pending the coming into force of such special agreements referred to in Article 43 as in the opinion of the Security Council enable it to begin the exercise of its responsibilities under Article 42 , the parties to the Four-Nation Declaration , signed at Moscow , 30 October 1943 , and France , shall , in accordance with the provisions of paragraph 5 of that Declaration , consult with one another and as 
 occasion requires with other Members of the United Nations with a view to such joint action on behalf of the Organization as may be necessary for the purpose of maintaining international peace and security . Article 107 Nothing in the present Charter shall invalidate or preclude action , in relation to any state which during the Second World War has been an enemy of any signatory to the present Charter , taken or authorized as a result of that war by the Governments having responsibility for such action . CHAPTER XVIII AMENDMENTS Article 108 Amendments to the present Charter shall come into force for all Members of the United Nations when they have been adopted by a vote of two thirds of the members of the General Assembly and ratified in accordance with their respective constitutional processes by two thirds of the Members of the United Nations , including all the permanent members of the Security Council . Article 109 1. A General Conference of the Members of the United Nations fo
 r the purpose of reviewing the present Charter may be held at a date and place to be fixed by a two-thirds vote of the members of the General Assembly and by a vote of any nine members of the Security Council . Each Member of the United Nations shall have one vote in the conference . 2. Any alteration of the present Charter recommended by a two-thirds vote of the conference shall take effect when ratified in accordance with their respective constitutional processes by two thirds of the Members of the United Nations including the permanent members of the Security Council . 3. If such a conference has not been held before the tenth annual session of the General Assembly following the coming into force of the present Charter , the proposal to call such a conference shall be placed on the agenda of that session of the General Assembly , and the conference shall be held if so decided by a majority vote of the members of the General Assembly and by a vote of any seven members of the Secur
 ity Council . CHAPTER XIX RATIFICATION AND SIGNATURE Article 110 1. The present Charter shall be ratified by the signatory states in accordance with their respective constitutional processes . 2. The shall be deposited with the Government of the Unite States of America , which shall notify a the signatory states of each deposit as well as the Secretary-General of the Organization when he has been appointed . 3. The present Charter shall come into force upon the deposit of by the Republic of China , France , the Union of Soviet Socialist , the United Kingdom of Great Britain and Northern Ireland , and the United States of America , and by a majority of the other signatory states . A protocol of the deposited shall thereupon be drawn up by the Government of the United States of America which shall communicate copies thereof to all the signatory states . 4. The states signatory to the present Charter which ratify it after it has come into force will become original Members of the Unite
 d Nations on the date of the deposit of their respective ratifications . Article 111 The present Charter , of which the Chinese , French , Russian , English , and Spanish texts are equally authentic , shall remain deposited in the archives of the Government of the United States of America . Duly certified copies thereof shall be transmitted by that Government to the Governments of the other signatory states . IN FAITH WHEREOF the representatives of the Governments of the United Nations have signed the present Charter . DONE at the city of San Francisco the twenty-sixth day of June , one thousand nine hundred and forty-five . 
\ No newline at end of file


[16/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Wind/139WindRiadhEt_ContrarotatingConverter_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Wind/139WindRiadhEt_ContrarotatingConverter_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Wind/139WindRiadhEt_ContrarotatingConverter_EN.txt.txt
new file mode 100644
index 0000000..a073ed1
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Wind/139WindRiadhEt_ContrarotatingConverter_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff Performance of a Contrarotating Small Wind Energy Converter 1. Introduction Wind energy has been shown to be one of the most feasible sources of renewable energy . It presents attractive opportunities to a wide range of people , including investors and entrepreneurs . The main goal of wind energy industry is to minimize the cost of wind energy in order to make it more competitive compared to other energy sources . How to reduce the cost of wind energy is a vital engineering challenge presented by the interlocking disciplines of aerodynamics , structure , control , electrical conversion , and electronics . In fact , technologies in these related areas are still under active research and development to achieve high efficiency and low cost . In the shadows of advancing multimegawatt wind turbines is another growing sector within this industry , the small wind turbines . Small wind energy converters ( SWECs ) for urban or rural applications range in size from a few hundred watts to 
 thousands of watts ( usually with a rated capacity of less than 100 kW ) and can be applied economically for a variety of power demands . These systems can be used in connection with an electricity transmission and distribution system , or in stand-alone applications that are not connected to the utility grid and are appropriate for homes , farms , or even entire communities . Investments in this sector are feasible not as stand-alone only , but as components of an integrated power-generating system that include various forms of energy resources . The main technical challenges for SWECs are the design of a system that has maximum efficiency in turbulent low speed winds ; ability to comply with both efficiency expectations and the requirements of grid utilities ; and have the minimum environmental and health impacts in terms of noise and vibration . Two facts characterize the urban environment for wind energy : lower annual mean wind speed ( AMWS ) compared to rural areas or to sea s
 hores and more turbulent flow . The low AMWS is related to the uneven ground created by buildings and other features of the urban landscape , which causes wind speeds to increase with height above the ground more slowly . The turbulent flow is a result of the wind interacting with landscape obstacles , the fact that applies extra stress on the turbine blades . The challenge is to develop wind turbines that operate at lower speeds and cope with the turbulent . The wind generating technology development is leading to improved performance and efficiency . Most wind turbines are single-rotor systems , which provide simplicity , reliability , and durability . Along the years , improvements have enhanced energy conversion efficiency of these single-rotor systems . For example , blades have better aerodynamic characteristics , gears with reduced noise have better torque transmission efficiency , and alternators have better electrical efficiency . However , despite these improvements , sing
 le-rotor systems are able to convert only a small fraction of the total wind stream energy into electrical energy . Moreover , such a system requires high wind velocity ( above 4 m/s ) which is not available in many places , a part from costal regions . This low velocity and seasonal winds imply a high cost of exploitation of wind energy . Thus , the challenge lies with the design of a wind generator which can operate at lower speeds and be used in a small-scale manner in remote and rural areas . This paper investigates the performance of the SWEC basing on wind tunnel tests . The paper is organized as follows . Section 2 discusses the contrarotating concepts and provides a literature review on the subject , while Section 3 presents the theory of rotor torque and power . Section 4 describes the wind tunnel experimental setup , and Section 5 presents the rotor performance results . Finally , conclusions are drawn in Section 6. 2. Contrarotating Blade System The prime mover in wind en
 ergy system is the wind turbine . One prevailing trend in wind turbine technology throughout the past couple of decades has been growth in the size of the rotor to realize the advantages of scale and the generally higher winds available at greater heights . Geometrically , consistent upscaling of blade length shows that the surface stresses at the blade surface , vibratory loads , and loading noise due to aerodynamical and gravitational loads grow in proportion to the length of the blade [ 1 ] . However , an alternative mean of overcoming the limitation of the efficiency of a single-rotor system without increasing the size of the rotor and consequently the stress on blades could be through the adoption of a dual-rotor ( contrarotating ) blade system . In addition , the acceptance of wind turbines by the public depends strongly on achieving low noise levels in operation , which largely depends on the level of stress on the blades . According to Betz theory , the maximum power that ca
 n be extracted from the wind is about 59 % of the available energy in the wind when the axial wind speed is reduced by two-thirds across a single rotor disc . However , practical wind turbines convert less than 40 % of the wind energy into electrical energy . Hence , nearly 60 % of the potential wind energy escapes without being harnessed . In reality , the energy in the wake behind a single rotor is not very small . Part of this energy may be extracted further by installing a second rotor in the wake . As the wake behind the first rotor is rotating in the opposite direction to the rotational direction of the rotor , the second rotor should rotate in the same direction as the wake in order to extract efficiently the available energy in the wake . The contrarotating system is a very old concept that was initially proposed more than 100 years ago . A friend of Betz who is sometimes described as the \u201c father of modern wind energy collection theory , \u201d Hans Honneff , wrote a book on
  the use of contrarotation , using two rotors one behind the other , driving the two halves of an electrical generator , therefore creating a true wind turbine [ 2 ] . Currently , the contra concept is used on airplanes , boats , and submarines to increase efficiency while eliminating the asymmetrical torque faced by conventional rotors . A dual-rotor system can be described as a system consisting of two rotors separated by an appropriate distance ( Figure 1 ) . One of the rotors is rotating in counterclockwise direction and the other in clockwise direction on the same axis . The relative size as well as the appropriate distance between the two rotors should be identified for best performance . Drawbacks of the dual-rotor system come from mechanical complexity based on the fact that in order to reverse direction of rotation of one rotor , a gearbox is required . This may increase weight or maintenance and spare parts cost for the system . Based on evidence in literature , aerodynami
 c research is poised between experimental and computational : either the wind turbine is studied experimentally in a wind tunnel , or the turbine is investigated computationally using methods that belong to the field of computational fluid dynamics ( CFD ) . The two are closely linked , and as progress is made in the development of more advanced computational fluid models , more comprehensive wind tunnel experimental data is required to validate the models . Experimental and computational research provide results for better understanding of the flow physics and enable investigation of wind energy performance , a requirement in order to adjust the design of wind turbines to the unique aerodynamic conditions in the environment . As with all methods of analysis , the CFD approach has limitations which are essentially related to turbulence modeling . Sumner et al . [ 3 ] review the development of CFD as a virtual , multiscale wind tunnel applied by the wind energy community from small t
 o large scale . Although the cost of a CFD analysis may be comparable to that of a wind tunnel experiment , we considered the wind tunnel experimental option for the current study emphasizing on the importance of transition to turbulence effects . Typically , wind tunnel tests overstate performance , and consumers will never see the performance measured in a wind tunnel . However , such tests are good indicators of performance . To our knowledge , only a limited number of wind tunnel studies can be found in literature [ 2 , 4 ] . In order to study the streamlines and obtain the detailed information of flow around the wind turbine , a flow visualization and velocity measurement are important . Investigation [ 5 ] has been carried out for this sake . Considerable improvements in the understanding of contrarotating wind turbine system can be achieved through proper instrumentation and experimental measurements . According to [ 6 ] , the maximum power that can be extracted from a dual-r
 otor system increases up to 64 % of the available energy . It continues to reach 66.7 % for an infinite number of rotors [ 7 ] . A contrarotating wind turbine equipped with two 500 kW turbines performed quite well at high wind speeds . The turbine can produce 43.5 % more annual energy than a single-rotor turbine of the same type . The performance of the system can be improved if it is operated for low wind speeds at the tip-speed ratio where a maximum Cp is obtained [ 8 ] . Research studies provide sufficient evidence to look closer at the concept of contrarotating system to eventually produce quantifiable comparisons to other turbines [ 9 , 10 ] . A smaller gear ratio is needed because of higher tip speeds achieved by smaller blade length in comparison with the conventional system in case of the same power output . Energy capture in the rotor holds the greatest potential for long-term reduction of the cost of wind energy . A feasibility study [ 11 ] provides sufficient evidence to 
 look closer at the concept of contrarotating to eventually produce quantifiable comparisons to other turbines . Their field tests showed that a dual-rotor turbine produces 43.5 % more annual energy than a single-rotor turbine of the same type . In addition , a smaller gear ratio is needed because of higher tip speeds achieved by smaller blade length in comparison with the conventional system in case of the same power output [ 12 ] . According to a field test demonstrated in California [ 13 ] , energy extraction from a wind turbine using contrarotating system increased by up to 40 % over an equivalent wind turbine with only one rotor . Power conversion efficiency was high at low rotor speeds , suggesting applicability of contrarotating turbines to large utility-scale wind turbines that rotate at 16\u201320 rpm . In addition , the bending stress on the supporting tower was reduced by the contrarotating system over the single-rotor system . This reduced bending stress results when the tor
 ques produced by two rotors counterbalance each other . The contrarotating SWEC clearly has a promise for wind energy , and after preliminary research and field studies [ 6\u201313 ] , it was decided to proceed with a small SWEC prototype for testing and evaluation . 3. Rotor Torque and Power The motion of any fluid can be derived from the basic physical principles of mass , momentum , and energy interchange . The torque responsible for power production of the wind turbine mostly arises due to the forces produced by interaction of blades with the wind . The output power \U0001d443 \U0001d447 from a turbine rotor and the wind kinetic energy per unit time \U0001d443 \U0001d44a are given as follows : \U0001d443 \U0001d447 = \U0001d447 \U0001d45a \U0001d443 � \U0001d714 , \U0001d44a = 1 2 \U0001d70c � \U0001d449 3 0 � \U0001d434 , ( 1 ) where \U0001d447 \U0001d45a is the mechanical torque at the turbine side , \U0001d714 is the angular rotation of the shaft , \U0001d70c is the air density at the hub height , \U0001d449 0 is the wind velocity , and \U0001d434 is the swept area of the blades . If momentum 
 equation is solved across an idealized control volume about the turbine rotor , it can be shown that the percentage of the total power available that can be extracted by a turbine is 16/27 or 59 % . This limit is known as the Betz limit . Therefore , the maximum power that a turbine can produce is expressed as follows [ 14 ] : \U0001d443 \U0001d44a = \ue080 1 6 1 2 7 \ue081 \ue080 2 \ue081 \U0001d70c � \U0001d449 3 0 � \U0001d434 . ( 2 ) Most turbines extract the maximum possible energy as defined above for lower wind speeds but gradually become less efficient as the on-coming wind speed increases and the flow condition across the blades approaches the stall condition . The rotor power coefficient \U0001d436 \U0001d45d is defined as the ratio between the rotor output power and the dynamic power of the air as shown in the following : \U0001d436 \U0001d45d = \U0001d443 \U0001d447 \U0001d443 \U0001d44a = \U0001d447 \U0001d45a � \U0001d714 \ue037 ( 1 / 2 ) \U0001d70c � \U0001d449 3 0 \ue038 � \U0001d434 . ( 3 ) The power coefficient is a nonlinear function of the tip speed-ratio \U0001d706 , which depends on the wind
  velocity and the rotation speed of the shaft \U0001d449 \U0001d706 = T i p \U0001d449 0 = \U0001d45f � \U0001d714 \U0001d449 0 , ( 4 ) where \U0001d45f is the rotor radius . The rotor power coefficient is regarded as the energy transformation efficiency . Note that \U0001d436 \U0001d45d is usually precomputed based on the theoretically expected performance of the turbine system . The wind turbine mechanical characteristics are described by the following equation ( where the turbine rotor friction is ignored ) : \U0001d447 \U0001d45a \u2212 \U0001d447 \U0001d454 = \U0001d43d \U0001d451 \U0001d714 \U0001d451 \U0001d461 , ( 5 ) where \U0001d447 \U0001d454 is the load torque , and \U0001d43d is the turbine inertia moment . The incoming wind flow rate should be equal to the outgoing flow rate to satisfy the mass conservation law if a control volume around a turbine is assumed . The outgoing wind-speed distribution and its direction strongly determine the turbine efficiency . Figure 2 shows the geometry of the stream tube through the disk . Neglecting fluid drags , the power extracted from the air stream can be w
 ritten as 1 \U0001d443 = ( 6 ) where \U0001d449 , and \U0001d449 are the flow velocity components along the axis of the stream tube . The power coefficient is obtained by nondimensionalizing the above power equation as \U0001d436 , ( 7 ) where \U0001d44e is the axial induction factor . 4. Wind Tunnel Experimental Setup In this section , laboratory measurement techniques are discussed ; however , some of the methods used are conventional and require little elaboration . 4.1 . Wind Tunnel Facility An open-return type wind tunnel is used in the present study . A contrarotating model 3-blade wind turbine was placed in the boundary-layer wind tunnel with the goal of studying power performance , turbulence effect , and flow visualization . Figure 3 shows the schematic of the wind tunnel experimental setup where the contraction and test sections are on the right hand side , and the motor and fan are in the left hand side . Air enters the fan from the laboratory through a large gate covered by a filter , held by wire me
 shes . The air flow is driven by a propulsion system made of an axial fan to provide the dynamic pressure for compensating viscous losses . There are smooth glass walls on both sides of the tunnel , and access is possible through the plywood ceiling and floor . Any large obstruction placed within a wind tunnel will alter the characteristics of the flow to some degree . The wind tunnel is capable of generating wind speeds up to 30 m/s . This suction type wind tunnel has a cross-section of 0.61 m width by 0.9 m height . The tunnel has a working ( test ) section of length 3.6 m. As the test section is the narrowest part of the circuit , it is also the part where the air velocity is the highest and , therefore , by Bernoulli\u2019s principle , where the pressure is the lowest . The main distinguishing feature of this wind tunnel is that it was designed to produce a low level of turbulence in the test section . Power for the tunnel comes from a three-phase AC motor of 30 hp at 1800 rpm with
  a maximum speed of 1170 rpm , driving a 10-bladed fan of 54 inches diameter with blade setting of 23� , mounted in a cylindrical steel casing . To minimise noise and vibration , the casing is supported on rubber shock mounts and is connected by flexible seals to the tunnel on either side . The air speed does not change as the air passes through the fan . The rotational speed of the fan is controlled by a regulated magnetic field and solid-state power supply . In order to control the ambient turbulence level , turbulence manipulators are placed upstream of the rotor , including a fine mesh screen and an aluminum honeycomb section . Smoothing is provided by the fine mesh screen . The honeycomb plays the role of a flow straightener . When the wind turbine is stopped , the mean velocity over the center portion of the wind tunnel is uniform and almost steady . 4.2 . Instrumentation A small model SWEC with two blade sets of 23 cm diameter and a varying distance between the blade sets of
  7\u201354 cm has been built and tested over a range of operating conditions . In order to introduce some degree of uniformity into the way in which users of the wind tunnel record their data , an instrumentation system to measure and display a number of variables that are normally required for all experiments was installed . Two guide rails were used to hold the SWEC inside wind tunnel floor along the centreline using a steel mounting system . The steel mounting system ensured that the system did not move during testing . Measuring sensors were mounted at different locations of the setup . The upwind and downwind velocities are measured by pitot tubes , which use Bernoulli\u2019s principle to convert pressure to velocity readings . The tubes are attached to 2 sensors to convert pressure in volts to velocities in m/s . For measuring the rotational speed of the rotor , two infrared detector and emitter units ( photogate sensors ) were used . They were mounted behind the rotor . Measurement
  depends largely on a data acquisition system utilizing electronic measuring to read instantaneous power produced by the generating system , as various parameters are varied on the turbine or in the environment . The parameters varied include the distance between the two sets of blades , blade profiles , number of blades , wind speeds , and size ratios . To accomplish the objective of this test , three aspects of experimental setup are needed : mechanical , electrical , and measurement software . All sensors are powered , grounded , and connected to the data acquisition board . All wires are shielded for protection against noise . Measurements are monitored directly and instantaneously in the Graphical User Interface ( GUI ) of LabView . The user enters numerical values of the blade distance , blade pitch , and blade diameter for the front and back and the relative humidity and temperature . The circuit has 5 sets of measurements on both the front and the back of the generating syst
 em . The voltages are measured directly from the potentiometers ; these are the total voltages of the circuits . The currents are obtained by measuring the voltages from fixed resistors and dividing that by the resistance . The power is the product of the voltage and the calculated current . The rpm signals go through a frequency measurement tool in LabView and are then multiplied by 60 to obtain the angular velocity in revolutions per minute ( rpm ) . All lines of measurements are connected to the National Instruments Data Acquisition Board NIDAQ USB-6210 . Each line is connected to an analog pin which is fed into the LabView program with a USB connection to the computer . At the beginning of the measurement process , all sensors were checked and calibrated . The pitot tubes are corrected by the offset values to give zero when there is no wind in the tunnel . When starting the program , a path is requested for an Excel file to record the data . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bact/311bacte\" url=\"http:__vue.org.uk_carlos.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/baiw/312baiwc\" url=\"http:__www.sleafordtownfc.co.uk_archives_archived_game.asp?MatchID=89&Season=2002_03.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bays/313baysn\" url=\"http:__www.portscathoholidays.co.uk_ShowDetails.asp?id=96.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bbgk/314bbgkl\" url=\"http:__www.homezonenews.org.uk_news_news_detail.asp?nid=22.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bbgl/315bbglz\" url=\"http:__www.benhs.org.uk_anex.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bcke/316bcked\" url=\"http:__www.fancy-rats.co.uk_information_guides_guides.php?subject=ratsthatbite.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bcnk/317bcnko\" url=\"http:__www.ombudsman.org.uk_improving_services_selected_cases_PCA_sc9903_c682b.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bdac/318bdacj\" url=\"http:__www.mml.cam.ac.uk_call_translation_toolkit_6_.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bdat/252bdat\" url=\"http:__www.snh.org.uk_nnr-scotland_news_detail.asp?newsID=79.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bdrl/319bdrlk\" url=\"http:__www.mubs.mdx.ac.uk_Conferences_BPCSR05_submission.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bdsv/320bdsvh\" url=\"http:__www.herts24.co.uk_flatfiles_paulpearcetributes.aspx.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/belc/321belcb\" url=\"http:__www.blackpresence.co.uk_phpBB2_viewtopic.php?t=97.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/beoe/322beoeg\" url=\"http:__www.viploan.co.uk_article_Mortgages-1212.shtml.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bfqb/323bfqbt\" url=\"http:__www.brainbashers.co.uk_droodlesprev.asp.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bgck/324bgckh\" url=\"http:__www.photonics.org.uk_newsletter_NoticeBoard.php.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bgnd/325bgndn\" url=\"http:__www.snh.org.uk_calendar_jul.asp.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bgpw/326bgpwt\" url=\"http:__www.cv-library.co.uk_localjobs_Northamptonshire_jobs-in-Brackley.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bgvt/327bgvth\" url=\"https:__secure.bfi.org.uk_features_ultimatefilm_chart_details.php?ranking=65.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bhdt/328bhdtl\" url=\"http:__eurocomms.co.uk_online_pr_online_pr.ehtml?o=1647.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bhtu/329bhtur\" url=\"http:__www.inverness-courier.co.uk_news_fullstory.php_aid_809_Tackling_human_organ_donation_dilemma_.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bhvh/330bhvhh\" url=\"http:__www.sscs.bham.ac.uk_phsi_eating_bmi.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/biec/331bieca\" url=\"http:__www.ombudsman.org.uk_improving_services_selected_cases_HSC_IC0107_pt1-e2242.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bjcq/332bjcqp\" url=\"http:__www.bfice.org.uk_index.asp?contentid=21&menuid=21.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bjyh/333bjyhd\" url=\"http:__www.ebe.org.uk_ccn.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bjzd/334bjzdi\" url=\"http:__www.northumberland.gov.uk_vg_text_northpen.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bkeb/335bkebq\" url=\"http:__www.vam.ac.uk_res_cons_research_research_reports_1992_theatre_museum_index.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bknb/336bknbk\" url=\"http:__easyweb.easynet.co.uk_jim.shead_River-Arun.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bksu/337bksuh\" url=\"http:__www.lawson-cruttenden.co.uk_conveyancing.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bkvb/338bkvbo\" url=\"http:__www.hamradio.co.uk_acatalog_Vert_Arno_Ant.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bkxn/339bkxng\" url=\"http:__travel.independent.co.uk_europe_article1192096.ece.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/blun/340blunb\" url=\"http:__www.nsbapty.co.uk_Supp-Samp.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bmbk/341bmbka\" url=\"http:__www.shakespeare-country.co.uk_swt.aspx?&cp=.._swt_&cg=_&sim=&id=487&pagetype=27.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bmwr/342bmwro\" url=\"http:__www.surf4wine.co.uk_Eben_Sadie.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bnaq/343bnaqt\" url=\"http:__jobsearch.localgov.monster.co.uk_getjob.asp?JobID=46663258&AVSDM=2006%2D08%2D10+09%3A45%3A00&Logo=0&sort=cp&pg=1.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bnqe/253bnqe\" url=\"http:__www.ecodyfi.org.uk_commfirstactionplanpr.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bofv/254bofv\" url=\"http:__www.sefton.gov.uk_page&3630.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bopl/344boplu\" url=\"http:__www.dw-perspective.org.uk_dwboard_messages_112.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/botu/345botuf\" url=\"http:__www.landforsale-investment.org.uk_Plot-Sales.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bova/346bovad\" url=\"http:__www.expertcardirectory.co.uk_car-leasing-jamjar.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bped/347bpedf\" url=\"http:__www.nta.nhs.uk_news_020624.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bpjo/348bpjoh\" url=\"http:__www1.city.ac.uk_law_lawpages_Victim_Support.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bpsf/349bpsfu\" url=\"http:__www.industrialnetworking.co.uk_mag_v7-2_p7.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bqbn/350bqbnl\" url=\"http:__www.cedr.co.uk_index.php?location=_news_archive_20040628.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bqcc/351bqccv\" url=\"http:__www.idler.co.uk_archives_?page_id=18.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bqsw/352bqswt\" url=\"http:__www.donhost.co.uk_support_index.pl?page=mailboxes.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bqxq/353bqxqv\" url=\"http:__backstage.bbc.co.uk_news_archives_2005_11_backstagebbccou_2.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/brcu/255brcu\" url=\"http:__www.motheratwork.co.uk_Health_default.asp?article=135.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/brjy/354brjyh\" url=\"http:__www.learningexperience.org.uk_learning_first.php.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bsaj/355bsajb\" url=\"http:__www.chortle.co.uk_edfest2006_terrysaunders.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bspm/356bspmu\" url=\"http:__www.cps.gov.uk_legal_section21_chapter_f.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bssg/357bssga\" url=\"http:__www.tropicalfishcentre.co.uk_Plants.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/btab/358btabi\" url=\"http:__www.bba.org.uk_bba_jsp_polopoly.jsp?d=155&a=493.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/btbb/359btbbg\" url=\"http:__www.burpham.surrey.sch.uk_potter.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bthk/360bthkw\" url=\"http:__www.ttrb.ac.uk_viewArticle.aspx?categoryId=14542&taggingType=4&contentId=11208.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bthx/361bthxs\" url=\"http:__www.rvrcd.co.uk_catalogue_walker_walkerreviews.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/btlm/362btlmk\" url=\"http:__www.birdtours.co.uk_tripreports_Spain_andalucia6_and-oct-03.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/btsm/363btsmp\" url=\"http:__www.trainingservicesindex.co.uk_newsletter_aug04.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/btsx/364btsxl\" url=\"http:__www.mediaweek.co.uk_search_index.cfm?fuseaction=details&nNewsID=560539.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/btvt/365btvti\" url=\"http:__www.i-dj.co.uk_artists_artistspage.php?ID=204&page=3.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bukc/366bukcu\" url=\"http:__www.ukpages.freewire.co.uk_buying-property-continent.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bvai/367bvaif\" url=\"http:__www.uservision.co.uk_usability_articles_print_wud.asp.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bvha/368bvhab\" url=\"http:__agrifor.ac.uk_browse_cabi_3736cbd2e5895cf49854f8d70494bae7.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bvqz/369bvqzk\" url=\"http:__www.elsham.pwp.blueyonder.co.uk_cx500_oil_pump_.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bvxo/370bvxom\" url=\"http:__www.schools.co.uk_index.php?name=News&file=article&sid=34.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bwal/371bwale\" url=\"http:__www.poptel.org.uk_scgn_articles_9902_inbrief.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bwvh/372bwvhf\" url=\"http:__www.tameside.gov.uk_tmbc6_cycling_withoutmycar.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bycu/373bycul\" url=\"http:__www.mubs.mdx.ac.uk_Staff_Personal_pages_Ifan1_Booth_Notebooks.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/byeg/374byegq\" url=\"http:__www.thehealthierlife.co.uk_article_3603_reduce-cancer-reoccurrence.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bykk/375bykkm\" url=\"http:__jobs.leaddiscovery.co.uk_job.aspx?jid=11535&cd=1.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bynx/376bynxp\" url=\"http:__www.syscom.plc.uk_solutions_distrib.asp.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bytv/377bytvq\" url=\"http:__personalfinance.iii.co.uk_articles_articledisplay.jsp?section=Banking&article_id=64923.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bzam/378bzamy\" url=\"http:__www.evolutec.co.uk_06_chairman.asp?thesub=6.0.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bzfk/379bzfki\" url=\"http:__www.truststfc.co.uk_meeting_27_09_2006.php.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bzic/380bzicn\" url=\"http:__www.macintyrecharity.org.uk_transition_personal_experiences_michael.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bziz/381bzizf\" url=\"http:__union.ic.ac.uk_scc_icsf_library_library_history_library_history_3.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bzlq/382bzlqz\" url=\"http:__www.socialistunitynetwork.co.uk_news_g8jepps.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bzor/383bzors\" url=\"http:__www.weirdwiltshire.co.uk_250703.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bzsx/384bzsxn\" url=\"http:__www.tsha.nhs.uk_modernising-healthcare-in-trent_the-local-supervising-authority-midwifery_lsa-guidelines_maternal-deaths.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/bzxm/385bzxmk\" url=\"http:__www.buildingproductexpert.co.uk_ExpandedEntries_expandedentry.asp?cid=212046&cname=Mark+Simpkin+Ltd&frmBPE=&frmCD=N&mopt=dpe&dpid=2302.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cahe/386cahel\" url=\"http:__www.siba.co.uk_about.asp.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cbfp/387cbfpd\" url=\"https:__secure.advanceperformance.co.uk_acatalog_Men_s_Wave_Nirvana_2_Mizuno_Running_Shoes.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cbuk/388cbukt\" url=\"http:__www.deafnessresearch.org.uk_?lid=1944&tmpl=ddmainprint.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cbxv/389cbxvh\" url=\"http:__www.port.ac.uk_departments_services_campusenvironment_.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cckm/390cckmj\" url=\"http:__www.dillington.co.uk_day_course_details.asp?ED=Arts+and+Crafts&offset=66.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/ccnl/256ccnl\" url=\"http:__www.esporta.co.uk_Clubs_Mids+%26+East+Anglia_Oxford_Promotions_Member%27s+Forum_!+!_CLASS_Advert_DBID_17ea4c66d7bd2c0aeb4513c89cb01afd.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cdfc/391cdfcq\" url=\"http:__www.fst.rdg.ac.uk_news-archive-2004-11.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cdng/392cdngg\" url=\"http:__www.aslib.co.uk_training_careers_9.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cdqv/393cdqvl\" url=\"http:__www.ncl.ac.uk_undergraduate_course_A106_profile_Can-I-spend-time-on-an-elective.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cedi/394cedie\" url=\"http:__www.incomesdata.co.uk_europe_duediligence.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cegc/395cegcr\" url=\"http:__www.pennine.demon.co.uk_NPC_1982_MEXICOSP.HTM.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/celp/396celpi\" url=\"http:__www.hsl.gov.uk_publications_car.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cest/397cesta\" url=\"http:__www.baronage.co.uk_bphtm-01_const-02.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cewp/398cewpo\" url=\"http:__www.assureweb.co.uk_public_Main.asp?Params=65C5B21F70C4D12078C6116FD0FD01ED50A0B7BBFBEBBDC7F85DB3C8C41964AFCF3977972B54AAC68E8AE50A7AB1888C6DEE8379864B7E79F21CA9025DF7DD55D14C83960FBA06562FFBA3B67013B5558FE96AD7.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cfec/399cfecj\" url=\"http:__www.myleedsjobs.co.uk_jobdetails-11834.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cfuh/400cfuhe\" url=\"http:__www.paperairplanes.co.uk_orplan.php.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cfvq/401cfvqe\" url=\"http:__www.eca.ac.uk_tacitus_news.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cfwj/402cfwje\" url=\"http:__www.casino-avenue.co.uk_2004_06_duuuuh.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cfwt/403cfwtc\" url=\"http:__www.dwp.gov.uk_lifeevent_penret_penreform_5_reg.asp.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cfxp/404cfxpi\" url=\"http:__www.framearch.co.uk_projects_T5_excavation.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cgbz/405cgbzt\" url=\"http:__www.redcross.org.uk_section.asp?id=49633.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cgel/406cgelv\" url=\"http:__www.forestforum.org.uk_jobs_forestsmonitor2001.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cgjx/407cgjxf\" url=\"http:__www.twickenham-museum.org.uk_kids_detail.asp?ContentID=189.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cgzy/257cgzy\" url=\"http:__www.offthetelly.co.uk_interviews_markwright.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/chns/408chnsk\" url=\"http:__www.sitcom.co.uk_tlc_characters.shtml.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/chny/409chnyq\" url=\"http:__www.aberdeen-grampian.co.uk_whiskycountry_ess_walk.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cicu/410cicug\" url=\"http:__www.shipleygreenparty.org.uk_sgpnewsarticle20051222a.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cipr/411ciprs\" url=\"http:__www.employment-solicitors.co.uk_Employer1.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/citx/412citxq\" url=\"http:__www.princessquare.co.uk_news_Food_Sounds_So_Good_at_DArcys_.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cjia/413cjiau\" url=\"http:__www.sweetsforu.co.uk_shipping.php.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cjjc/414cjjcl\" url=\"http:__www.e.volve.org.uk_Listings.aspx?index=387&item=2929.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cjqc/415cjqcv\" url=\"http:__www.verko.co.uk_product.aspx?catno=53&prod=HCAA6241.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cjta/416cjtan\" url=\"http:__www.romancesouthwest.co.uk_main_en_att-provider-ROMA_6913.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cjto/417cjton\" url=\"http:__www.art-works.org.uk_artworks_z030703b.shtml.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cjtu/418cjtub\" url=\"http:__www.jr2.ox.ac.uk_bandolier_booth_miscellaneous_wristgang.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cjwg/419cjwgx\" url=\"http:__www.changingdiabetes.co.uk_view.asp?ID=92.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/ckcu/420ckcux\" url=\"http:__www.lashelmets.co.uk_las%20new%20bionix%20page.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/ckiv/421ckivb\" url=\"http:__www.cb-com.co.uk_listgen.asp?layout=results-brief.asp&page=37&sql=&sortup=sorttitle&bookstatus=OK.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/ckju/422ckjul\" url=\"http:__www.wildlifetrust.org.uk_cheshire_proj_harvest_survey.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/ckrq/423ckrqy\" url=\"http:__www.enemydown.co.uk_clancomments.php?id=35113.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/ckym/424ckymf\" url=\"http:__www.scis.org.uk_search_menu_new.asp.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/clez/425clezz\" url=\"http:__www.uk-wholesaler.co.uk_softbook_clickbankmembership_clickbankprotector.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/clsl/426clslk\" url=\"http:__www.blewa.co.uk_project5_teachers_T5-0-1.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cmna/427cmnal\" url=\"http:__www.politicalwizard.co.uk_administration_childsocnew_index.php?category=campaigns&c=i&uid=2130.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cnos/428cnosx\" url=\"http:__www.kent-ccc.co.uk_news_story.php?id=660.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cnoz/429cnozg\" url=\"http:__beehive.thisisexeter.co.uk_default.asp?WCI=SiteHome&ID=9908&PageID=56638.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cnqw/430cnqwk\" url=\"http:__bookshop.blackwell.co.uk_jsp_id_0340894342_Divine_Madness.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cntu/431cntup\" url=\"http:__www.newble.co.uk_chalmers_innes.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cody/432codym\" url=\"http:__www.searchenginespy.co.uk_article0027.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/coip/433coipg\" url=\"http:__www.scottishcorpus.ac.uk_corpus_search_document.php?documentid=1211.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cplj/434cpljw\" url=\"http:__www.peterhead.org.uk_familyheritage_forum_topic.asp?TOPIC_ID=26&.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cpnw/435cpnwa\" url=\"http:__www.frenchhouserestoration.co.uk_franceproperty150to200_propertyandhousesforsalelimousinabn0509263.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cqha/436cqham\" url=\"http:__www.rcpsych.ac.uk_college_faculties_liaison_documents_servicedevelopment_managerialfacilities.aspx.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cqng/437cqngo\" url=\"http:__www.tlchm.bris.ac.uk_safety_various_rass_kmweb_safety_msds.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cqrd/438cqrdh\" url=\"http:__www.msabritain.co.uk_index.php?id=23&L=3&article=13.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/crkw/439crkwd\" url=\"http:__www.frontier.net.uk_FAQsearch.asp?search_strFields=strMetaKeywords&search_strType=FAQS&search_strAreaNo=1053,2011&strKeyword=PS2006_4_3.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/crsr/440crsrx\" url=\"http:__www.itreviews.co.uk_games_g232.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/csah/441csahf\" url=\"http:__www.petergasston.co.uk_2002_09_to-quote-the-four-seasons.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/csct/442csctp\" url=\"http:__www.kevinmayhew.co.uk_Mobile_default.aspx?group_id=16538&article_id=21979.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/csmw/443csmwd\" url=\"http:__www.ceac.aston.ac.uk_research_staff_jpf_papers_paper26_index.php.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/csqc/444csqcw\" url=\"http:__www.all-energy.co.uk_newsletter45.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/csxj/445csxju\" url=\"http:__socialistworker.org.uk_article.php?article_id=8138.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/ctcu/446ctcuk\" url=\"http:__www.setdanceteacher.co.uk_newmarketmez.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/ctfd/447ctfdn\" url=\"http:__www.ebi.ac.uk_interpro_DisplayIproEntry?ac=IPR002824.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/ctgo/448ctgol\" url=\"http:__www.garthyfog.co.uk_mawddach_valley.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cuua/258cuua\" url=\"http:__www.swcc.org.uk_caving_expeditions_jura05_jura_circ1.php.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cvbw/449cvbwe\" url=\"http:__www.oca-online.co.uk_viewnews.cfm?news_id=177.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cvgk/450cvgkz\" url=\"http:__www.chichester.co.uk_mk4CustomPages_CustomPage.aspx?PageID=24163&sectionID=4585.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cvqe/451cvqer\" url=\"http:__www.webstar.co.uk_~musnews_news_search.php?search=&start=12080.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cwbp/452cwbpn\" url=\"http:__www.linc4info.org.uk_cms_pages_sitemap.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cwcq/453cwcqx\" url=\"http:__www.ccp4.ac.uk_courses_IUCr2005_index.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cwde/454cwdel\" url=\"http:__www.perceptive-engineering.co.uk_html_training.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cwhb/455cwhbc\" url=\"http:__www.lanpac.co.uk_csi.php.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cwma/456cwmaf\" url=\"http:__www.chisenhale.org.uk_html_files_501_project_info.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cwzw/457cwzwc\" url=\"http:__www.ilnpictures.co.uk_showpage.asp?showdocumentid=196.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cxnk/458cxnko\" url=\"http:__www.starlink.rl.ac.uk_star_docs_sun232.htx_node17.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cxxy/459cxxyp\" url=\"http:__www.publications.parliament.uk_pa_cm199900_cmhansrd_vo000405_debtext_00405-07.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/cyyc/460cyyci\" url=\"http:__www.dwalker.pwp.blueyonder.co.uk_Fasti%20V.2_p.%20278%20PRESBYTERY%20OF%20PENPONT%20p.%20672.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/czqo/461czqoj\" url=\"http:__www.wessingtoncryogenics.co.uk_serv01.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/czve/462czvea\" url=\"http:__www.aylesburyvale.gov.uk_avdc_content_index.jsp?contentid=1999276669.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/dagx/463dagxc\" url=\"http:__www.stratford-upon-avon.co.uk_static_481.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/danu/464danug\" url=\"http:__www.synergygroup.co.uk_office-support-recruitment_.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/daqq/465daqqs\" url=\"http:__www.sexshop365.co.uk_catalog_product_info.php?products_id=2981.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/daws/466dawss\" url=\"http:__www.honda-racing.co.uk_fourwheels_formula1_article.asp?a=1327.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/dbbq/467dbbqw\" url=\"http:__www.bioinformatics.leeds.ac.uk_~david_docs_api_javax_swing_JSplitPane.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/dbhi/468dbhia\" url=\"http:__www.ateonline.co.uk_60_66_67_articles_7335.php.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/dbjz/469dbjza\" url=\"http:__www.iae.co.uk_news_designedforthejob.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/dbzi/470dbzic\" url=\"http:__www.oaa.org.uk_Case_Studies_studies_Ford3_Ford3.asp.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/dccp/471dccpz\" url=\"http:__www.jamesgourmetcoffee.co.uk_product.php?xProd=21&xSec=22.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/dcon/472dcont\" url=\"http:__www.uservision.co.uk_usability_aboutus_usability_aboutus.asp.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/dcqe/473dcqen\" url=\"http:__www.learnenglish.org.uk_crazyworld_series2_crazyworld_story.asp?latestchapter=12&subarea=11.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/dcsp/474dcspt\" url=\"http:__www.guysherratt.co.uk_pages_searchdetails.asp?ID=776.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/dddx/475dddxm\" url=\"http:__www.menshealthforum.org.uk_userpage1.cfm?item_id=1913.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/ddqu/476ddquk\" url=\"http:__www.schoolhouse.org.uk_law_not_enrolled.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/deck/477decku\" url=\"http:__www.womenintothenetwork.co.uk_page_calendar_archive_article.cfm?articleId=52.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/decp/478decpr\" url=\"http:__www.lathes.co.uk_beaver_page5.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/dehf/479dehfj\" url=\"http:__www.hasslefreeminiatures.co.uk_rules.php.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/deqg/480deqgv\" url=\"http:__www.newworknetwork.org.uk_modules_event_viewevent.php?eveid=109.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/derj/481derjo\" url=\"http:__www.cheatgenius.co.uk_cheats_641_Gamecube-cheats_Gamecube-(hardware)-Cheats.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/dfnm/482dfnmt\" url=\"http:__www.omega.co.uk_ppt_pptsc.asp?ref=LE902.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/dfoo/483dfoor\" url=\"http:__www.fulcrum-anglican.org.uk_forum_poster.cfm?sort=creatasc&poster=101.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/dfvl/484dfvlp\" url=\"http:__www.la-hq.org.uk_directory_prof_issues_blreview.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/dfzy/485dfzyc\" url=\"http:__www.billyarmstrong.co.uk_050613_public_urged_to_back_london's_olympic_2012_bid.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/dgmo/486dgmor\" url=\"http:__chat.dailymail.co.uk_dailymail_threadnonInd.jsp?forum=106&thread=9757638&message=11724737.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/dgti/487dgtix\" url=\"http:__www.thebookpeople.co.uk_webapp_wcs_stores_servlet_product_10001_10051_20553_100_10012_10010_category_10010.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/dgws/488dgwsx\" url=\"http:__www.bjhc.co.uk_news_industry_2005_ind505016.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/dgzt/489dgztd\" url=\"http:__www.uea.ac.uk_eas_events_litfestspr04.shtml.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/dheb/490dhebu\" url=\"http:__www.kimberry.co.uk_Dotnetlectures_Index.aspx.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/dheg/491dhegq\" url=\"http:__www.lathes.co.uk_wolfjahnmiller_page2.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/dhem/492dhemc\" url=\"http:__www.humanism.org.uk_site_cms_newsarticleview.asp?article=2173.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/dhow/493dhowh\" url=\"http:__www.anweb.co.uk_l_04_c3_c3a10.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/eizz/259eizz\" url=\"http:__www.royalhigh.edin.sch.uk_content_subject_modernlanguages_course_s1s2.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/fodw/260fodw\" url=\"http:__www.law.warwick.ac.uk_ltj_4-1m.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/fvms/261fvms\" url=\"http:__www.citizenshipfoundation.org.uk_main_news.php?n20.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/guzs/262guzs\" url=\"http:__www.arnside-online.co.uk_care.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/hhvw/263hhvw\" url=\"http:__www.oarsport.co.uk_products_leatherman_micra.php.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/hkay/264hkay\" url=\"http:__www.bnp.org.uk_columnists_docdiary2.php?docId=103.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/hzbi/265hzbi\" url=\"http:__www.familiesonline.co.uk_article_articleview_1733_1_153.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/kbmr/266kbmr\" url=\"http:__www.ladybird-survey.pwp.blueyonder.co.uk_P_mugo.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/kjkq/267kjkq\" url=\"http:__www.free-internet.co.uk_email_sendmail.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/kzix/268kzix\" url=\"http:__www.cs.bham.ac.uk_resources_ums_PythonDoc_api_threads.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/lfwv/269lfwv\" url=\"http:__www.thisismoney.co.uk_news_columnists_article.html?in_article_id=405873&in_page_id=50002.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/ljrm/270ljrm\" url=\"http:__www.learningservices.gcal.ac.uk_synergy_03_scwbl.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/lptf/271lptf\" url=\"http:__www.nisp.co.uk_pooled_articles_BF_NEWSART_view.asp?Q=BF_NEWSART_95582.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/lqyu/272lqyu\" url=\"http:__www.mountainsoftware.co.uk_printpage.asp?REF=_group.asp.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/mail/273mail\" url=\"http:__www.dba.org.uk_aboutdba_chriswood.asp.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/mpxl/274mpxl\" url=\"http:__www.surefish.co.uk_culture_books_0804_110804_food_books.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/mtps/275mtps\" url=\"http:__www.newtsnni.gov.uk_actionplan_04b.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/mtqb/276mtqb\" url=\"http:__www.hummersknott.org.uk_Stud_Res_Info_Tec_Info_ICT_KS3_databases_relational_databases.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/ndi\"/247ndi\" url=\"http:__www.stockportmbc.gov.uk_secondary_offerton_pages_SchemesofWork_KS4_skillswl.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/odv\"/248odv\" url=\"http:__www.find-me-a-gift.co.uk_gifts-for-men_unusual-gadgets_mood-light-tile.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/omzz/277omzz\" url=\"http:__www.swindon-speedway.co.uk_modules.php?op=modload&name=News&file=article&sid=175&mode=thread&order=0&thold=0.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/oojb/278oojb\" url=\"http:__www.engender.org.uk_justice.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/opab/279opab\" url=\"http:__www.heros.org.uk_home_sub.asp?page=2.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/orle/280orle\" url=\"http:__www.bsg.org.uk_clinical_prac_mar_05_mar05_08.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/osor/281osor\" url=\"http:__www.cdp.bham.ac.uk_About_CDP_methods.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/pabf/282pabf\" url=\"https:__www.cambs-police.co.uk_caminfo_blueprint_articles.asp?ID=807.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/pkbn/283pkbn\" url=\"http:__www.hmso.gov.uk_legislation_scotland_acts2002_20017--b.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/pxnz/284pxnz\" url=\"http:__www.dog-pictures.co.uk_dog-pictures_shiba_inu.shtml.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/qsym/285qsym\" url=\"http:__www.strawbale-building.co.uk_index.php?page=faq.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/qvpo/286qvpo\" url=\"http:__company.monster.co.uk_londonunderuk_tfl_our_careers.asp.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/rfk\"/249rfk\" url=\"http:__www.environment.bham.ac.uk_extindex.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/rfkm/287rfkm\" url=\"http:__www.artshole.co.uk_exhibitions_Aug%2006%2004_James%20Cauty.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/rivm/288rivm\" url=\"http:__www.sincuser.f9.co.uk_050_lastwrd.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/rkwj/289rkwj\" url=\"http:__www.ocdaction.org.uk_skin-picking.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/rlwz/290rlwz\" url=\"http:__www.greenparty.org.uk_news_2033.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/rnce/291rnce\" url=\"http:__www.lel.ed.ac.uk_linguist_issues_17_17-229.html.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/sbdw/292sbdw\" url=\"http:__sccplugins.sheffield.gov.uk_press_news_aRelease.asp?akey=2026&Mon=01_07_2004.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/sbqp/293sbqp\" url=\"http:__www.lpt.nhs.uk_service5.php.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/seaz/294seaz\" url=\"http:__www.panos.org.uk_resources_reportdetails.asp?id=1039.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/smar/295smar\" url=\"http:__www.arctech.co.uk_siemens_hosted_exchange.htm.txt"
----------------------------------------------------------------------

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/"opennlp-similarity/src/test/resources/style_recognizer/txt/sqdo/296sqdo\" url=\"http:__gimbo.org.uk_archives_2006_01_chomsky_intervi.html.txt"
----------------------------------------------------------------------


[38/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/46FictBalzacH_Goriot_Ia_EN.txt.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/46FictBalzacH_Goriot_Ia_EN.txt.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/46FictBalzacH_Goriot_Ia_EN.txt.txt
new file mode 100644
index 0000000..071e72d
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/Fict/46FictBalzacH_Goriot_Ia_EN.txt.txt
@@ -0,0 +1,2 @@
+
+\ufeff FATHER GORIOT To the great and illustrious Geoffroy Saint-Hilaire , a token of admiration for his works and genius . DE BALZAC . FATHER GORIOT Mme . Vauquer ( _nee_ de Conflans ) is an elderly person , who for the past forty years has kept a lodging-house in the Rue Nueve-Sainte-Genevieve , in the district that lies between the Latin Quarter and the Faubourg Saint-Marcel . Her house ( known in the neighborhood as the _Maison Vauquer_ ) receives men and women , old and young , and no word has ever been breathed against her respectable establishment ; but , at the same time , it must be said that as a matter of fact no young woman has been under her roof for thirty years , and that if a young man stays there for any length of time it is a sure sign that his allowance must be of the slenderest . In 1819 , however , the time when this drama opens , there was an almost penniless young girl among Mme . Vauquer 's boarders . That word drama has been somewhat discredited of late ; it ha
 s been overworked and twisted to strange uses in these days of dolorous literature ; but it must do service again here , not because this story is dramatic in the restricted sense of the word , but because some tears may perhaps be shed _intra et extra muros_ before it is over . Will any one without the walls of Paris understand it ? It is open to doubt . The only audience who could appreciate the results of close observation , the careful reproduction of minute detail and local color , are dwellers between the heights of Montrouge and Montmartre , in a vale of crumbling stucco watered by streams of black mud , a vale of sorrows which are real and joys too often hollow ; but this audience is so accustomed to terrible sensations , that only some unimaginable and well-neigh impossible woe could produce any lasting impression there . Now and again there are tragedies so awful and so grand by reason of the complication of virtues and vices that bring them about , that egotism and selfis
 hness are forced to pause and are moved to pity ; but the impression that they receive is like a luscious fruit , soon consumed . Civilization , like the car of Juggernaut , is scarcely stayed perceptibly in its progress by a heart less easy to break than the others that lie in its course ; this also is broken , and Civilization continues on her course triumphant . And you , too , will do the like ; you who with this book in your white hand will sink back among the cushions of your armchair , and say to yourself , " Perhaps this may amuse me . " You will read the story of Father Goriot 's secret woes , and , dining thereafter with an unspoiled appetite , will lay the blame of your insensibility upon the writer , and accuse him of exaggeration , of writing romances . Ah ! once for all , this drama is neither a fiction nor a romance ! _All is true_ , --so true , that every one can discern the elements of the tragedy in his own house , perhaps in his own heart . The lodging-house is Mm
 e . Vauquer 's own property . It is still standing in the lower end of the Rue Nueve-Sainte-Genevieve , just where the road slopes so sharply down to the Rue de l'Arbalete , that wheeled traffic seldom passes that way , because it is so stony and steep . This position is sufficient to account for the silence prevalent in the streets shut in between the dome of the Pantheon and the dome of the Val-de-Grace , two conspicuous public buildings which give a yellowish tone to the landscape and darken the whole district that lies beneath the shadow of their leaden-hued cupolas . In that district the pavements are clean and dry , there is neither mud nor water in the gutters , grass grows in the chinks of the walls . The most heedless passer-by feels the depressing influences of a place where the sound of wheels creates a sensation ; there is a grim look about the houses , a suggestion of a jail about those high garden walls . A Parisian straying into a suburb apparently composed of lodging
 -houses and public institutions would see poverty and dullness , old age lying down to die , and joyous youth condemned to drudgery . It is the ugliest quarter of Paris , and , it may be added , the least known . But , before all things , the Rue Nueve-Sainte-Genevieve is like a bronze frame for a picture for which the mind cannot be too well prepared by the contemplation of sad hues and sober images . Even so , step by step the daylight decreases , and the cicerone 's droning voice grows hollower as the traveler descends into the Catacombs . The comparison holds good ! Who shall say which is more ghastly , the sight of the bleached skulls or of dried-up human hearts ? The front of the lodging-house is at right angles to the road , and looks out upon a little garden , so that you see the side of the house in section , as it were , from the Rue Nueve-Sainte-Genevieve . Beneath the wall of the house front there lies a channel , a fathom wide , paved with cobble-stones , and beside it 
 runs a graveled walk bordered by geraniums and oleanders and pomegranates set in great blue and white glazed earthenware pots . Access into the graveled walk is afforded by a door , above which the words MAISON VAUQUER may be read , and beneath , in rather smaller letters , " _Lodgings for both sexes , etc. _ " During the day a glimpse into the garden is easily obtained through a wicket to which a bell is attached . On the opposite wall , at the further end of the graveled walk , a green marble arch was painted once upon a time by a local artist , and in this semblance of a shrine a statue representing Cupid is installed ; a Parisian Cupid , so blistered and disfigured that he looks like a candidate for one of the adjacent hospitals , and might suggest an allegory to lovers of symbolism . The half-obliterated inscription on the pedestal beneath determines the date of this work of art , for it bears witness to the widespread enthusiasm felt for Voltaire on his return to Paris in 1777
  : " Whoe'er thou art , thy master see ; He is , or was , or ought to be . " At night the wicket gate is replaced by a solid door . The little garden is no wider than the front of the house ; it is shut in between the wall of the street and the partition wall of the neighboring house . A mantle of ivy conceals the bricks and attracts the eyes of passers-by to an effect which is picturesque in Paris , for each of the walls is covered with trellised vines that yield a scanty dusty crop of fruit , and furnish besides a subject of conversation for Mme . Vauquer and her lodgers ; every year the widow trembles for her vintage . A straight path beneath the walls on either side of the garden leads to a clump of lime-trees at the further end of it ; _line_-trees , as Mme . Vauquer persists in calling them , in spite of the fact that she was a de Conflans , and regardless of repeated corrections from her lodgers . The central space between the walls is filled with artichokes and rows of pyram
 id fruit-trees , and surrounded by a border of lettuce , pot-herbs , and parsley . Under the lime-trees there are a few green-painted garden seats and a wooden table , and hither , during the dog-days , such of the lodgers as are rich enough to indulge in a cup of coffee come to take their pleasure , though it is hot enough to roast eggs even in the shade . The house itself is three stories high , without counting the attics under the roof . It is built of rough stone , and covered with the yellowish stucco that gives a mean appearance to almost every house in Paris . There are five windows in each story in the front of the house ; all the blinds visible through the small square panes are drawn up awry , so that the lines are all at cross purposes . At the side of the house there are but two windows on each floor , and the lowest of all are adorned with a heavy iron grating . Behind the house a yard extends for some twenty feet , a space inhabited by a happy family of pigs , poultry
  , and rabbits ; the wood-shed is situated on the further side , and on the wall between the wood-shed and the kitchen window hangs the meat-safe , just above the place where the sink discharges its greasy streams . The cook sweeps all the refuse out through a little door into the Rue Nueve-Sainte-Genevieve , and frequently cleanses the yard with copious supplies of water , under pain of pestilence . The house might have been built on purpose for its present uses . Access is given by a French window to the first room on the ground floor , a sitting-room which looks out upon the street through the two barred windows already mentioned . Another door opens out of it into the dining-room , which is separated from the kitchen by the well of the staircase , the steps being constructed partly of wood , partly of tiles , which are colored and beeswaxed . Nothing can be more depressing than the sight of that sitting-room . The furniture is covered with horse hair woven in alternate dull and 
 glossy stripes . There is a round table in the middle , with a purplish-red marble top , on which there stands , by way of ornament , the inevitable white china tea-service , covered with a half-effaced gilt network . The floor is sufficiently uneven , the wainscot rises to elbow height , and the rest of the wall space is decorated with a varnished paper , on which the principal scenes from _Telemaque_ are depicted , the various classical personages being colored . The subject between the two windows is the banquet given by Calypso to the son of Ulysses , displayed thereon for the admiration of the boarders , and has furnished jokes these forty years to the young men who show themselves superior to their position by making fun of the dinners to which poverty condemns them . The hearth is always so clean and neat that it is evident that a fire is only kindled there on great occasions ; the stone chimney-piece is adorned by a couple of vases filled with faded artificial flowers impris
 oned under glass shades , on either side of a bluish marble clock in the very worst taste . The first room exhales an odor for which there is no name in the language , and which should be called the _odeur de pension_ . The damp atmosphere sends a chill through you as you breathe it ; it has a stuffy , musty , and rancid quality ; it permeates your clothing ; after-dinner scents seem to be mingled in it with smells from the kitchen and scullery and the reek of a hospital . It might be possible to describe it if some one should discover a process by which to distil from the atmosphere all the nauseating elements with which it is charged by the catarrhal exhalations of every individual lodger , young or old . Yet , in spite of these stale horrors , the sitting-room is as charming and as delicately perfumed as a boudoir , when compared with the adjoining dining-room . The paneled walls of that apartment were once painted some color , now a matter of conjecture , for the surface is incr
 usted with accumulated layers of grimy deposit , which cover it with fantastic outlines . A collection of dim-ribbed glass decanters , metal discs with a satin sheen on them , and piles of blue-edged earthenware plates of Touraine ware cover the sticky surfaces of the sideboards that line the room . In a corner stands a box containing a set of numbered pigeon-holes , in which the lodgers ' table napkins , more or less soiled and stained with wine , are kept . Here you see that indestructible furniture never met with elsewhere , which finds its way into lodging-houses much as the wrecks of our civilization drift into hospitals for incurables . You expect in such places as these to find the weather-house whence a Capuchin issues on wet days ; you look to find the execrable engravings which spoil your appetite , framed every one in a black varnished frame , with a gilt beading round it ; you know the sort of tortoise-shell clock-case , inlaid with brass ; the green stove , the Argand l
 amps , covered with oil and dust , have met your eyes before . The oilcloth which covers the long table is so greasy that a waggish _externe_ will write his name on the surface , using his thumb-nail as a style . The chairs are broken-down invalids ; the wretched little hempen mats slip away from under your feet without slipping away for good ; and finally , the foot-warmers are miserable wrecks , hingeless , charred , broken away about the holes . It would be impossible to give an idea of the old , rotten , shaky , cranky , worm-eaten , halt , maimed , one-eyed , rickety , and ramshackle condition of the furniture without an exhaustive description , which would delay the progress of the story to an extent that impatient people would not pardon . The red tiles of the floor are full of depressions brought about by scouring and periodical renewings of color . In short , there is no illusory grace left to the poverty that reigns here ; it is dire , parsimonious , concentrated , threadb
 are poverty ; as yet it has not sunk into the mire , it is only splashed by it , and though not in rags as yet , its clothing is ready to drop to pieces . This apartment is in all its glory at seven o'clock in the morning , when Mme . Vauquer 's cat appears , announcing the near approach of his mistress , and jumps upon the sideboards to sniff at the milk in the bowls , each protected by a plate , while he purrs his morning greeting to the world . A moment later the widow shows her face ; she is tricked out in a net cap attached to a false front set on awry , and shuffles into the room in her slipshod fashion . She is an oldish woman , with a bloated countenance , and a nose like a parrot 's beak set in the middle of it ; her fat little hands ( she is as sleek as a church rat ) and her shapeless , slouching figure are in keeping with the room that reeks of misfortune , where hope is reduced to speculate for the meanest stakes . Mme . Vauquer alone can breathe that tainted air withou
 t being disheartened by it . Her face is as fresh as a frosty morning in autumn ; there are wrinkles about the eyes that vary in their expression from the set smile of a ballet-dancer to the dark , suspicious scowl of a discounter of bills ; in short , she is at once the embodiment and interpretation of her lodging-house , as surely as her lodging-house implies the existence of its mistress . You can no more imagine the one without the other , than you can think of a jail without a turnkey . The unwholesome corpulence of the little woman is produced by the life she leads , just as typhus fever is bred in the tainted air of a hospital . The very knitted woolen petticoat that she wears beneath a skirt made of an old gown , with the wadding protruding through the rents in the material , is a sort of epitome of the sitting-room , the dining-room , and the little garden ; it discovers the cook , it foreshadows the lodgers--the picture of the house is completed by the portrait of its mist
 ress . Mme . Vauquer at the age of fifty is like all women who " have seen a deal of trouble . " She has the glassy eyes and innocent air of a trafficker in flesh and blood , who will wax virtuously indignant to obtain a higher price for her services , but who is quite ready to betray a Georges or a Pichegru , if a Georges or a Pichegru were in hiding and still to be betrayed , or for any other expedient that may alleviate her lot . Still , " she is a good woman at bottom , " said the lodgers who believed that the widow was wholly dependent upon the money that they paid her , and sympathized when they heard her cough and groan like one of themselves . What had M. Vauquer been ? The lady was never very explicit on this head . How had she lost her money ? " Through trouble , " was her answer . He had treated her badly , had left her nothing but her eyes to cry over his cruelty , the house she lived in , and the privilege of pitying nobody , because , so she was wont to say , she herse
 lf had been through every possible misfortune . Sylvie , the stout cook , hearing her mistress ' shuffling footsteps , hastened to serve the lodgers ' breakfasts . Beside those who lived in the house , Mme . Vauquer took boarders who came for their meals ; but these _externes_ usually only came to dinner , for which they paid thirty francs a month . At the time when this story begins , the lodging-house contained seven inmates . The best rooms in the house were on the first story , Mme . Vauquer herself occupying the least important , while the rest were let to a Mme . Couture , the widow of a commissary-general in the service of the Republic . With her lived Victorine Taillefer , a schoolgirl , to whom she filled the place of mother . These two ladies paid eighteen hundred francs a year . The two sets of rooms on the second floor were respectively occupied by an old man named Poiret and a man of forty or thereabouts , the wearer of a black wig and dyed whiskers , who gave out that 
 he was a retired merchant , and was addressed as M. Vautrin . Two of the four rooms on the third floor were also let--one to an elderly spinster , a Mlle . Michonneau , and the other to a retired manufacturer of vermicelli , Italian paste and starch , who allowed the others to address him as " Father Goriot . " The remaining rooms were allotted to various birds of passage , to impecunious students , who like " Father Goriot " and Mlle . Michonneau , could only muster forty-five francs a month to pay for their board and lodging . Mme . Vauquer had little desire for lodgers of this sort ; they ate too much bread , and she only took them in default of better . At that time one of the rooms was tenanted by a law student , a young man from the neighborhood of Angouleme , one of a large family who pinched and starved themselves to spare twelve hundred francs a year for him . Misfortune had accustomed Eugene de Rastignac , for that was his name , to work . He belonged to the number of youn
 g men who know as children that their parents ' hopes are centered on them , and deliberately prepare themselves for a great career , subordinating their studies from the first to this end , carefully watching the indications of the course of events , calculating the probable turn that affairs will take , that they may be the first to profit by them . But for his observant curiosity , and the skill with which he managed to introduce himself into the salons of Paris , this story would not have been colored by the tones of truth which it certainly owes to him , for they are entirely due to his penetrating sagacity and desire to fathom the mysteries of an appalling condition of things , which was concealed as carefully by the victim as by those who had brought it to pass . Above the third story there was a garret where the linen was hung to dry , and a couple of attics . Christophe , the man-of-all-work , slept in one , and Sylvie , the stout cook , in the other . Beside the seven inma
 tes thus enumerated , taking one year with another , some eight law or medical students dined in the house , as well as two or three regular comers who lived in the neighborhood . There were usually eighteen people at dinner , and there was room , if need be , for twenty at Mme . Vauquer 's table ; at breakfast , however , only the seven lodgers appeared . It was almost like a family party . Every one came down in dressing-gown and slippers , and the conversation usually turned on anything that had happened the evening before ; comments on the dress or appearance of the dinner contingent were exchanged in friendly confidence . These seven lodgers were Mme . Vauquer 's spoiled children . Among them she distributed , with astronomical precision , the exact proportion of respect and attention due to the varying amounts they paid for their board . One single consideration influenced all these human beings thrown together by chance . The two second-floor lodgers only paid seventy-two fra
 ncs a month . Such prices as these are confined to the Faubourg Saint-Marcel and the district between La Bourbe and the Salpetriere ; and , as might be expected , poverty , more or less apparent , weighed upon them all , Mme . Couture being the sole exception to the rule . The dreary surroundings were reflected in the costumes of the inmates of the house ; all were alike threadbare . The color of the men 's coats were problematical ; such shoes , in more fashionable quarters , are only to be seen lying in the gutter ; the cuffs and collars were worn and frayed at the edges ; every limp article of clothing looked like the ghost of its former self . The women 's dresses were faded , old-fashioned , dyed and re-dyed ; they wore gloves that were glazed with hard wear , much-mended lace , dingy ruffles , crumpled muslin fichus . So much for their clothing ; but , for the most part , their frames were solid enough ; their constitutions had weathered the storms of life ; their cold , hard 
 faces were worn like coins that have been withdrawn from circulation , but there were greedy teeth behind the withered lips . Dramas brought to a close or still in progress are foreshadowed by the sight of such actors as these , not the dramas that are played before the footlights and against a background of painted canvas , but dumb dramas of life , frost-bound dramas that sere hearts like fire , dramas that do not end with the actors ' lives . Mlle . Michonneau , that elderly young lady , screened her weak eyes from the daylight by a soiled green silk shade with a rim of brass , an object fit to scare away the Angel of Pity himself . Her shawl , with its scanty , draggled fringe , might have covered a skeleton , so meagre and angular was the form beneath it . Yet she must have been pretty and shapely once . What corrosive had destroyed the feminine outlines ? Was it trouble , or vice , or greed ? Had she loved too well ? Had she been a second-hand clothes dealer , a frequenter of 
 the backstairs of great houses , or had she been merely a courtesan ? Was she expiating the flaunting triumphs of a youth overcrowded with pleasures by an old age in which she was shunned by every passer-by ? Her vacant gaze sent a chill through you ; her shriveled face seemed like a menace . Her voice was like the shrill , thin note of the grasshopper sounding from the thicket when winter is at hand . She said that she had nursed an old gentleman , ill of catarrh of the bladder , and left to die by his children , who thought that he had nothing left . His bequest to her , a life annuity of a thousand francs , was periodically disputed by his heirs , who mingled slander with their persecutions . In spite of the ravages of conflicting passions , her face retained some traces of its former fairness and fineness of tissue , some vestiges of the physical charms of her youth still survived . M. Poiret was a sort of automaton . He might be seen any day sailing like a gray shadow along the
  walks of the Jardin des Plantes , on his head a shabby cap , a cane with an old yellow ivory handle in the tips of his thin fingers ; the outspread skirts of his threadbare overcoat failed to conceal his meagre figure ; his breeches hung loosely on his shrunken limbs ; the thin , blue-stockinged legs trembled like those of a drunken man ; there was a notable breach of continuity between the dingy white waistcoat and crumpled shirt frills and the cravat twisted about a throat like a turkey gobbler 's ; altogether , his appearance set people wondering whether this outlandish ghost belonged to the audacious race of the sons of Japhet who flutter about on the Boulevard Italien . What devouring kind of toil could have so shriveled him ? What devouring passions had darkened that bulbous countenance , which would have seemed outrageous as a caricature ? What had he been ? Well , perhaps he had been part of the machinery of justice , a clerk in the office to which the executioner sends in 
 his accounts , --so much for providing black veils for parricides , so much for sawdust , so much for pulleys and cord for the knife . Or he might have been a receiver at the door of a public slaughter-house , or a sub-inspector of nuisances . Indeed , the man appeared to have been one of the beasts of burden in our great social mill ; one of those Parisian Ratons whom their Bertrands do not even know by sight ; a pivot in the obscure machinery that disposes of misery and things unclean ; one of those men , in short , at sight of whom we are prompted to remark that , " After all , we cannot do without them . " Stately Paris ignores the existence of these faces bleached by moral or physical suffering ; but , then , Paris is in truth an ocean that no line can plumb . You may survey its surface and describe it ; but no matter how numerous and painstaking the toilers in this sea , there will always be lonely and unexplored regions in its depths , caverns unknown , flowers and pearls and
  monsters of the deep overlooked or forgotten by the divers of literature . The Maison Vauquer is one of these curious monstrosities . Two , however , of Mme . Vauquer 's boarders formed a striking contrast to the rest . There was a sickly pallor , such as is often seen in anaemic girls , in Mlle . Victorine Taillefer 's face ; and her unvarying expression of sadness , like her embarrassed manner and pinched look , was in keeping with the general wretchedness of the establishment in the Rue Nueve-Saint-Genevieve , which forms a background to this picture ; but her face was young , there was youthfulness in her voice and elasticity in her movements . This young misfortune was not unlike a shrub , newly planted in an uncongenial soil , where its leaves have already begun to wither . The outlines of her figure , revealed by her dress of the simplest and cheapest materials , were also youthful . There was the same kind of charm about her too slender form , her faintly colored face and l
 ight-brown hair , that modern poets find in mediaeval statuettes ; and a sweet expression , a look of Christian resignation in the dark gray eyes . She was pretty by force of contrast ; if she had been happy , she would have been charming . Happiness is the poetry of woman , as the toilette is her tinsel . If the delightful excitement of a ball had made the pale face glow with color ; if the delights of a luxurious life had brought the color to the wan cheeks that were slightly hollowed already ; if love had put light into the sad eyes , then Victorine might have ranked among the fairest ; but she lacked the two things which create woman a second time--pretty dresses and love-letters . A book might have been made of her story . Her father was persuaded that he had sufficient reason for declining to acknowledge her , and allowed her a bare six hundred francs a year ; he had further taken measures to disinherit his daughter , and had converted all his real estate into personalty , tha
 t he might leave it undivided to his son . Victorine 's mother had died broken-hearted in Mme . Couture 's house ; and the latter , who was a near relation , had taken charge of the little orphan . Unluckily , the widow of the commissary-general to the armies of the Republic had nothing in the world but her jointure and her widow 's pension , and some day she might be obliged to leave the helpless , inexperienced girl to the mercy of the world . The good soul , therefore , took Victorine to mass every Sunday , and to confession once a fortnight , thinking that , in any case , she would bring up her ward to be devout . She was right ; religion offered a solution of the problem of the young girl 's future . The poor child loved the father who refused to acknowledge her . Once every year she tried to see him to deliver her mother 's message of forgiveness , but every year hitherto she had knocked at that door in vain ; her father was inexorable . Her brother , her only means of communi
 cation , had not come to see her for four years , and had sent her no assistance ; yet she prayed to God to unseal her father 's eyes and to soften her brother 's heart , and no accusations mingled with her prayers . Mme . Couture and Mme . Vauquer exhausted the vocabulary of abuse , and failed to find words that did justice to the banker 's iniquitous conduct ; but while they heaped execrations on the millionaire , Victorine 's words were as gentle as the moan of the wounded dove , and affection found expression even in the cry drawn from her by pain . Eugene de Rastignac was a thoroughly southern type ; he had a fair complexion , blue eyes , black hair . In his figure , manner , and his whole bearing it was easy to see that he had either come of a noble family , or that , from his earliest childhood , he had been gently bred . If he was careful of his wardrobe , only taking last year 's clothes into daily wear , still upon occasion he could issue forth as a young man of fashion . 
 Ordinarily he wore a shabby coat and waistcoat , the limp black cravat , untidily knotted , that students affect , trousers that matched the rest of his costume , and boots that had been resoled . Vautrin ( the man of forty with the dyed whiskers ) marked a transition stage between these two young people and the others . He was the kind of man that calls forth the remark : " He looks a jovial sort ! " He had broad shoulders , a well-developed chest , muscular arms , and strong square-fisted hands ; the joints of his fingers were covered with tufts of fiery red hair . His face was furrowed by premature wrinkles ; there was a certain hardness about it in spite of his bland and insinuating manner . His bass voice was by no means unpleasant , and was in keeping with his boisterous laughter . He was always obliging , always in good spirits ; if anything went wrong with one of the locks , he would soon unscrew it , take it to pieces , file it , oil and clean and set it in order , and put 
 it back in its place again ; " I am an old hand at it , " he used to say . Not only so , he knew all about ships , the sea , France , foreign countries , men , business , law , great houses and prisons , --there was nothing that he did not know . If any one complained rather more than usual , he would offer his services at once . He had several times lent money to Mme . Vauquer , or to the boarders ; but , somehow , those whom he obliged felt that they would sooner face death than fail to repay him ; a certain resolute look , sometimes seen on his face , inspired fear of him , for all his appearance of easy good-nature . In the way he spat there was an imperturbable coolness which seemed to indicate that this was a man who would not stick at a crime to extricate himself from a false position . His eyes , like those of a pitiless judge , seemed to go to the very bottom of all questions , to read all natures , all feelings and thoughts . His habit of life was very regular ; he usually
  went out after breakfast , returning in time for dinner , and disappeared for the rest of the evening , letting himself in about midnight with a latch key , a privilege that Mme . Vauquer accorded to no other boarder . But then he was on very good terms with the widow ; he used to call her " mamma , " and put his arm round her waist , a piece of flattery perhaps not appreciated to the full ! The worthy woman might imagine this to be an easy feat ; but , as a matter of fact , no arm but Vautrin 's was long enough to encircle her . It was a characteristic trait of his generously to pay fifteen francs a month for the cup of coffee with a dash of brandy in it , which he took after dinner . Less superficial observers than young men engulfed by the whirlpool of Parisian life , or old men , who took no interest in anything that did not directly concern them , would not have stopped short at the vaguely unsatisfactory impression that Vautrin made upon them . He knew or guessed the concerns
  of every one about him ; but none of them had been able to penetrate his thoughts , or to discover his occupation . He had deliberately made his apparent good-nature , his unfailing readiness to oblige , and his high spirits into a barrier between himself and the rest of them , but not seldom he gave glimpses of appalling depths of character . He seemed to delight in scourging the upper classes of society with the lash of his tongue , to take pleasure in convicting it of inconsistency , in mocking at law and order with some grim jest worthy of Juvenal , as if some grudge against the social system rankled in him , as if there were some mystery carefully hidden away in his life . Mlle . Taillefer felt attracted , perhaps unconsciously , by the strength of the one man , and the good looks of the other ; her stolen glances and secret thoughts were divided between them ; but neither of them seemed to take any notice of her , although some day a chance might alter her position , and she 
 would be a wealthy heiress . For that matter , there was not a soul in the house who took any trouble to investigate the various chronicles of misfortunes , real or imaginary , related by the rest . Each one regarded the others with indifference , tempered by suspicion ; it was a natural result of their relative positions . Practical assistance not one could give , this they all knew , and they had long since exhausted their stock of condolence over previous discussions of their grievances . They were in something the same position as an elderly couple who have nothing left to say to each other . The routine of existence kept them in contact , but they were parts of a mechanism which wanted oil . There was not one of them but would have passed a blind man begging in the street , not one that felt moved to pity by a tale of misfortune , not one who did not see in death the solution of the all-absorbing problem of misery which left them cold to the most terrible anguish in others . Th
 e happiest of these hapless beings was certainly Mme . Vauquer , who reigned supreme over this hospital supported by voluntary contributions . For her , the little garden , which silence , and cold , and rain , and drought combined to make as dreary as an Asian _steppe_ , was a pleasant shaded nook ; the gaunt yellow house , the musty odors of a back shop had charms for her , and for her alone . Those cells belonged to her . She fed those convicts condemned to penal servitude for life , and her authority was recognized among them . Where else in Paris would they have found wholesome food in sufficient quantity at the prices she charged them , and rooms which they were at liberty to make , if not exactly elegant or comfortable , at any rate clean and healthy ? If she had committed some flagrant act of injustice , the victim would have borne it in silence . Such a gathering contained , as might have been expected , the elements out of which a complete society might be constructed . An
 d , as in a school , as in the world itself , there was among the eighteen men and women who met round the dinner table a poor creature , despised by all the others , condemned to be the butt of all their jokes . At the beginning of Eugene de Rastignac 's second twelvemonth , this figure suddenly started out into bold relief against the background of human forms and faces among which the law student was yet to live for another two years to come . This laughing-stock was the retired vermicelli-merchant , Father Goriot , upon whose face a painter , like the historian , would have concentrated all the light in his picture . How had it come about that the boarders regarded him with a half-malignant contempt ? Why did they subject the oldest among their number to a kind of persecution , in which there was mingled some pity , but no respect for his misfortunes ? Had he brought it on himself by some eccentricity or absurdity , which is less easily forgiven or forgotten than more serious de
 fects ? The question strikes at the root of many a social injustice . Perhaps it is only human nature to inflict suffering on anything that will endure suffering , whether by reason of its genuine humility , or indifference , or sheer helplessness . Do we not , one and all , like to feel our strength even at the expense of some one or of something ? The poorest sample of humanity , the street arab , will pull the bell handle at every street door in bitter weather , and scramble up to write his name on the unsullied marble of a monument . In the year 1813 , at the age of sixty-nine or thereabouts , " Father Goriot " had sold his business and retired--to Mme . Vauquer 's boarding house . When he first came there he had taken the rooms now occupied by Mme . Couture ; he had paid twelve hundred francs a year like a man to whom five louis more or less was a mere trifle . For him Mme . Vauquer had made various improvements in the three rooms destined for his use , in consideration of a ce
 rtain sum paid in advance , so it was said , for the miserable furniture , that is to say , for some yellow cotton curtains , a few chairs of stained wood covered with Utrecht velvet , several wretched colored prints in frames , and wall papers that a little suburban tavern would have disdained . Possibly it was the careless generosity with which Father Goriot allowed himself to be overreached at this period of his life ( they called him Monsieur Goriot very respectfully then ) that gave Mme . Vauquer the meanest opinion of his business abilities ; she looked on him as an imbecile where money was concerned . Goriot had brought with him a considerable wardrobe , the gorgeous outfit of a retired tradesman who denies himself nothing . Mme . Vauquer 's astonished eyes beheld no less than eighteen cambric-fronted shirts , the splendor of their fineness being enhanced by a pair of pins each bearing a large diamond , and connected by a short chain , an ornament which adorned the vermicelli
 -maker 's shirt front . He usually wore a coat of corn-flower blue ; his rotund and portly person was still further set off by a clean white waistcoat , and a gold chain and seals which dangled over that broad expanse . When his hostess accused him of being " a bit of a beau , " he smiled with the vanity of a citizen whose foible is gratified . His cupboards ( _ormoires_ , as he called them in the popular dialect ) were filled with a quantity of plate that he brought with him . The widow 's eyes gleamed as she obligingly helped him to unpack the soup ladles , table-spoons , forks , cruet-stands , tureens , dishes , and breakfast services--all of silver , which were duly arranged upon shelves , besides a few more or less handsome pieces of plate , all weighing no inconsiderable number of ounces ; he could not bring himself to part with these gifts that reminded him of past domestic festivals . " This was my wife 's present to me on the first anniversary of our wedding day , " he said
  to Mme . Vauquer , as he put away a little silver posset dish , with two turtle-doves billing on the cover . " Poor dear ! she spent on it all the money she had saved before we were married . Do you know , I would sooner scratch the earth with my nails for a living , madame , than part with that . But I shall be able to take my coffee out of it every morning for the rest of my days , thank the Lord ! I am not to be pitied . There 's not much fear of my starving for some time to come . " Finally , Mme . Vauquer 's magpie 's eye had discovered and read certain entries in the list of shareholders in the funds , and , after a rough calculation , was disposed to credit Goriot ( worthy man ) with something like ten thousand francs a year . From that day forward Mme . Vauquer ( _nee_ de Conflans ) , who , as a matter of fact , had seen forty-eight summers , though she would only own to thirty-nine of them--Mme . Vauquer had her own ideas . Though Goriot 's eyes seemed to have shrunk in th
 eir sockets , though they were weak and watery , owing to some glandular affection which compelled him to wipe them continually , she considered him to be a very gentlemanly and pleasant-looking man . Moreover , the widow saw favorable indications of character in the well-developed calves of his legs and in his square-shaped nose , indications still further borne out by the worthy man 's full-moon countenance and look of stupid good-nature . This , in all probability , was a strongly-build animal , whose brains mostly consisted in a capacity for affection . His hair , worn in _ailes de pigeon_ , and duly powdered every morning by the barber from the Ecole Polytechnique , described five points on his low forehead , and made an elegant setting to his face . Though his manners were somewhat boorish , he was always as neat as a new pin and he took his snuff in a lordly way , like a man who knows that his snuff-box is always likely to be filled with maccaboy , so that when Mme . Vauquer 
 lay down to rest on the day of M. Goriot 's installation , her heart , like a larded partridge , sweltered before the fire of a burning desire to shake off the shroud of Vauquer and rise again as Goriot . She would marry again , sell her boarding-house , give her hand to this fine flower of citizenship , become a lady of consequence in the quarter , and ask for subscriptions for charitable purposes ; she would make little Sunday excursions to Choisy , Soissy , Gentilly ; she would have a box at the theatre when she liked , instead of waiting for the author 's tickets that one of her boarders sometimes gave her , in July ; the whole Eldorado of a little Parisian household rose up before Mme . Vauquer in her dreams . Nobody knew that she herself possessed forty thousand francs , accumulated _sou by sou_ , that was her secret ; surely as far as money was concerned she was a very tolerable match . " And in other respects , I am quite his equal , " she said to herself , turning as if to 
 assure herself of the charms of a form that the portly Sylvie found moulded in down feathers every morning . For three months from that day Mme . Veuve Vauquer availed herself of the services of M. Goriot 's coiffeur , and went to some expense over her toilette , expense justifiable on the ground that she owed it to herself and her establishment to pay some attention to appearances when such highly-respectable persons honored her house with their presence . She expended no small amount of ingenuity in a sort of weeding process of her lodgers , announcing her intention of receiving henceforward none but people who were in every way select . If a stranger presented himself , she let him know that M. Goriot , one of the best known and most highly-respected merchants in Paris , had singled out her boarding-house for a residence . She drew up a prospectus headed MAISON VAUQUER , in which it was asserted that hers was " _one of the oldest and most highly recommended boarding-houses in the
  Latin Quarter_ . " " From the windows of the house , " thus ran the prospectus , " there is a charming view of the Vallee des Gobelins ( so there is--from the third floor ) , and a _beautiful_ garden , _extending_ down to _an avenue of lindens_ at the further end . " Mention was made of the bracing air of the place and its quiet situation . It was this prospectus that attracted Mme . la Comtesse de l'Ambermesnil , a widow of six and thirty , who was awaiting the final settlement of her husband 's affairs , and of another matter regarding a pension due to her as the wife of a general who had died " on the field of battle . " On this Mme . Vauquer saw to her table , lighted a fire daily in the sitting-room for nearly six months , and kept the promise of her prospectus , even going to some expense to do so . And the Countess , on her side , addressed Mme . Vauquer as " my dear , " and promised her two more boarders , the Baronne de Vaumerland and the widow of a colonel , the late Comt
 e de Picquoisie , who were about to leave a boarding-house in the Marais , where the terms were higher than at the Maison Vauquer . Both these ladies , moreover , would be very well to do when the people at the War Office had come to an end of their formalities . " But Government departments are always so dilatory , " the lady added . After dinner the two widows went together up to Mme . Vauquer 's room , and had a snug little chat over some cordial and various delicacies reserved for the mistress of the house . Mme . Vauquer 's ideas as to Goriot were cordially approved by Mme . de l'Ambermesnil ; it was a capital notion , which for that matter she had guessed from the very first ; in her opinion the vermicelli maker was an excellent man . " Ah ! my dear lady , such a well-preserved man of his age , as sound as my eyesight--a man who might make a woman happy ! " said the widow . The good-natured Countess turned to the subject of Mme . Vauquer 's dress , which was not in harmony wit
 h her projects . " You must put yourself on a war footing , " said she . After much serious consideration the two widows went shopping together--they purchased a hat adorned with ostrich feathers and a cap at the Palais Royal , and the Countess took her friend to the Magasin de la Petite Jeannette , where they chose a dress and a scarf . Thus equipped for the campaign , the widow looked exactly like the prize animal hung out for a sign above an a la mode beef shop ; but she herself was so much pleased with the improvement , as she considered it , in her appearance , that she felt that she lay under some obligation to the Countess ; and , though by no means open-handed , she begged that lady to accept a hat that cost twenty francs . The fact was that she needed the Countess ' services on the delicate mission of sounding Goriot ; the countess must sing her praises in his ears . Mme . de l'Ambermesnil lent herself very good-naturedly to this manoeuvre , began her operations , and succe
 eded in obtaining a private interview ; but the overtures that she made , with a view to securing him for herself , were received with embarrassment , not to say a repulse . She left him , revolted by his coarseness . " My angel , " said she to her dear friend , " you will make nothing of that man yonder . He is absurdly suspicious , and he is a mean curmudgeon , an idiot , a fool ; you would never be happy with him . " After what had passed between M. Goriot and Mme . de l'Ambermesnil , the Countess would no longer live under the same roof . She left the next day , forgot to pay for six months ' board , and left behind her wardrobe , cast-off clothing to the value of five francs . Eagerly and persistently as Mme . Vauquer sought her quondam lodger , the Comtesse de l'Ambermesnil was never heard of again in Paris . The widow often talked of this deplorable business , and regretted her own too confiding disposition . As a matter of fact , she was as suspicious as a cat ; but she was 
 like many other people , who cannot trust their own kin and put themselves at the mercy of the next chance comer--an odd but common phenomenon , whose causes may readily be traced to the depths of the human heart . Perhaps there are people who know that they have nothing more to look for from those with whom they live ; they have shown the emptiness of their hearts to their housemates , and in their secret selves they are conscious that they are severely judged , and that they deserve to be judged severely ; but still they feel an unconquerable craving for praises that they do not hear , or they are consumed by a desire to appear to possess , in the eyes of a new audience , the qualities which they have not , hoping to win the admiration or affection of strangers at the risk of forfeiting it again some day . Or , once more , there are other mercenary natures who never do a kindness to a friend or a relation simply because these have a claim upon them , while a service done to a stra
 nger brings its reward to self-love . Such natures feel but little affection for those who are nearest to them ; they keep their kindness for remoter circles of acquaintance , and show most to those who dwell on its utmost limits . Mme . Vauquer belonged to both these essentially mean , false , and execrable classes . " If I had been there at the time , " Vautrin would say at the end of the story , " I would have shown her up , and that misfortune would not have befallen you . I know that kind of phiz ! " Like all narrow natures , Mme . Vauquer was wont to confine her attention to events , and did not go very deeply into the causes that brought them about ; she likewise preferred to throw the blame of her own mistakes on other people , so she chose to consider that the honest vermicelli maker was responsible for her misfortune . It had opened her eyes , so she said , with regard to him . As soon as she saw that her blandishments were in vain , and that her outlay on her toilette was
  money thrown away , she was not slow to discover the reason of his indifference . It became plain to her at once that there was _some other attraction_ , to use her own expression . In short , it was evident that the hope she had so fondly cherished was a baseless delusion , and that she would " never make anything out of that man yonder , " in the Countess ' forcible phrase . The Countess seemed to have been a judge of character . Mme . Vauquer 's aversion was naturally more energetic than her friendship , for her hatred was not in proportion to her love , but to her disappointed expectations . The human heart may find here and there a resting-place short of the highest height of affection , but we seldom stop in the steep , downward slope of hatred . Still , M. Goriot was a lodger , and the widow 's wounded self-love could not vent itself in an explosion of wrath ; like a monk harassed by the prior of his convent , she was forced to stifle her sighs of disappointment , and to gul
 p down her craving for revenge . Little minds find gratification for their feelings , benevolent or otherwise , by a constant exercise of petty ingenuity . The widow employed her woman 's malice to devise a system of covert persecution . She began by a course of retrenchment--various luxuries which had found their way to the table appeared there no more . " No more gherkins , no more anchovies ; they have made a fool of me ! " she said to Sylvie one morning , and they returned to the old bill of fare . The thrifty frugality necessary to those who mean to make their way in the world had become an inveterate habit of life with M. Goriot . Soup , boiled beef , and a dish of vegetables had been , and always would be , the dinner he liked best , so Mme . Vauquer found it very difficult to annoy a boarder whose tastes were so simple . He was proof against her malice , and in desperation she spoke to him and of him slightingly before the other lodgers , who began to amuse themselves at his
  expense , and so gratified her desire for revenge . Towards the end of the first year the widow 's suspicions had reached such a pitch that she began to wonder how it was that a retired merchant with a secure income of seven or eight thousand livres , the owner of such magnificent plate and jewelry handsome enough for a kept mistress , should be living in her house . Why should he devote so small a proportion of his money to his expenses ? Until the first year was nearly at an end , Goriot had dined out once or twice every week , but these occasions came less frequently , and at last he was scarcely absent from the dinner-table twice a month . It was hardly expected that Mme . Vauquer should regard the increased regularity of her boarder 's habits with complacency , when those little excursions of his had been so much to her interest . She attributed the change not so much to a gradual diminution of fortune as to a spiteful wish to annoy his hostess . It is one of the most detestab
 le habits of a Liliputian mind to credit other people with its own malignant pettiness . Unluckily , towards the end of the second year , M. Goriot 's conduct gave some color to the idle talk about him . He asked Mme . Vauquer to give him a room on the second floor , and to make a corresponding reduction in her charges . Apparently , such strict economy was called for , that he did without a fire all through the winter . Mme . Vauquer asked to be paid in advance , an arrangement to which M. Goriot consented , and thenceforward she spoke of him as " Father Goriot . " What had brought about this decline and fall ? Conjecture was keen , but investigation was difficult . Father Goriot was not communicative ; in the sham countess ' phrase he was " a curmudgeon . " Empty-headed people who babble about their own affairs because they have nothing else to occupy them , naturally conclude that if people say nothing of their doings it is because their doings will not bear being talked about ; 
 so the highly respectable merchant became a scoundrel , and the late beau was an old rogue . Opinion fluctuated . Sometimes , according to Vautrin , who came about this time to live in the Maison Vauquer , Father Goriot was a man who went on 'Change and _dabbled_ ( to use the sufficiently expressive language of the Stock Exchange ) in stocks and shares after he had ruined himself by heavy speculation . Sometimes it was held that he was one of those petty gamblers who nightly play for small stakes until they win a few francs . A theory that he was a detective in the employ of the Home Office found favor at one time , but Vautrin urged that " Goriot was not sharp enough for one of that sort . " There were yet other solutions ; Father Goriot was a skinflint , a shark of a money-lender , a man who lived by selling lottery tickets . He was by turns all the most mysterious brood of vice and shame and misery ; yet , however vile his life might be , the feeling of repulsion which he aroused
  in others was not so strong that he must be banished from their society--he paid his way . Besides , Goriot had his uses , every one vented his spleen or sharpened his wit on him ; he was pelted with jokes and belabored with hard words . The general consensus of opinion was in favor of a theory which seemed the most likely ; this was Mme . Vauquer 's view . According to her , the man so well preserved at his time of life , as sound as her eyesight , with whom a woman might be very happy , was a libertine who had strange tastes . These are the facts upon which Mme . Vauquer 's slanders were based . Early one morning , some few months after the departure of the unlucky Countess who had managed to live for six months at the widow 's expense , Mme . Vauquer ( not yet dressed ) heard the rustle of a silk dress and a young woman 's light footstep on the stair ; some one was going to Goriot 's room . He seemed to expect the visit , for his door stood ajar . The portly Sylvie presently cam
 e up to tell her mistress that a girl too pretty to be honest , " dressed like a goddess , " and not a speck of mud on her laced cashmere boots , had glided in from the street like a snake , had found the kitchen , and asked for M. Goriot 's room . Mme . Vauquer and the cook , listening , overheard several words affectionately spoken during the visit , which lasted for some time . When M. Goriot went downstairs with the lady , the stout Sylvie forthwith took her basket and followed the lover-like couple , under pretext of going to do her marketing . " M. Goriot must be awfully rich , all the same , madame , " she reported on her return , " to keep her in such style . Just imagine it ! There was a splendid carriage waiting at the corner of the Place de l'Estrapade , and _she_ got into it . " While they were at dinner that evening , Mme . Vauquer went to the window and drew the curtain , as the sun was shining into Goriot 's eyes . " You are beloved of fair ladies , M. Goriot--the sun
  seeks you out , " she said , alluding to his visitor . " _Peste ! _ you have good taste ; she was very pretty . " " That was my daughter , " he said , with a kind of pride in his voice , and the rest chose to consider this as the fatuity of an old man who wishes to save appearances . A month after this visit M. Goriot received another . The same daughter who had come to see him that morning came again after dinner , this time in evening dress . The boarders , in deep discussion in the dining-room , caught a glimpse of a lovely , fair-haired woman , slender , graceful , and much too distinguished-looking to be a daughter of Father Goriot 's . " Two of them ! " cried the portly Sylvie , who did not recognize the lady of the first visit . A few days later , and another young lady--a tall , well-moulded brunette , with dark hair and bright eyes--came to ask for M. Goriot . " Three of them ! " said Sylvie . Then the second daughter , who had first come in the morning to see her father ,
  came shortly afterwards in the evening . She wore a ball dress , and came in a carriage . " Four of them ! " commented Mme . Vauquer and her plump handmaid . Sylvie saw not a trace of resemblance between this great lady and the girl in her simple morning dress who had entered her kitchen on the occasion of her first visit . At that time Goriot was paying twelve hundred francs a year to his landlady , and Mme . Vauquer saw nothing out of the common in the fact that a rich man had four or five mistresses ; nay , she thought it very knowing of him to pass them off as his daughters . She was not at all inclined to draw a hard-and-fast line , or to take umbrage at his sending for them to the Maison Vauquer ; yet , inasmuch as these visits explained her boarder 's indifference to her , she went so far ( at the end of the second year ) as to speak of him as an " ugly old wretch . " When at length her boarder declined to nine hundred francs a year , she asked him very insolently what he to
 ok her house to be , after meeting one of these ladies on the stairs . Father Goriot answered that the lady was his eldest daughter . " So you have two or three dozen daughters , have you ? " said Mme . Vauquer sharply . " I have only two , " her boarder answered meekly , like a ruined man who is broken in to all the cruel usage of misfortune . Towards the end of the third year Father Goriot reduced his expenses still further ; he went up to the third story , and now paid forty-five francs a month . He did without snuff , told his hairdresser that he no longer required his services , and gave up wearing powder . When Goriot appeared for the first time in this condition , an exclamation of astonishment broke from his hostess at the color of his hair--a dingy olive gray . He had grown sadder day by day under the influence of some hidden trouble ; among all the faces round the table , his was the most woe-begone . There was no longer any doubt . Goriot was an elderly libertine , whose 
 eyes had only been preserved by the skill of the physician from the malign influence of the remedies necessitated by the state of his health . The disgusting color of his hair was a result of his excesses and of the drugs which he had taken that he might continue his career . The poor old man 's mental and physical condition afforded some grounds for the absurd rubbish talked about him . When his outfit was worn out , he replaced the fine linen by calico at fourteen _sous_ the ell . His diamonds , his gold snuff-box , watch-chain and trinkets , disappeared one by one . He had left off wearing the corn-flower blue coat , and was sumptuously arrayed , summer as well as winter , in a coarse chestnut-brown coat , a plush waistcoat , and doeskin breeches . He grew thinner and thinner ; his legs were shrunken , his cheeks , once so puffed out by contented bourgeois prosperity , were covered with wrinkles , and the outlines of the jawbones were distinctly visible ; there were deep furrows 
 in his forehead . In the fourth year of his residence in the Rue Neuve-Sainte-Genevieve he was no longer like his former self . The hale vermicelli manufacturer , sixty-two years of age , who had looked scarce forty , the stout , comfortable , prosperous tradesman , with an almost bucolic air , and such a brisk demeanor that it did you good to look at him ; the man with something boyish in his smile , had suddenly sunk into his dotage , and had become a feeble , vacillating septuagenarian . The keen , bright blue eyes had grown dull , and faded to a steel-gray color ; the red inflamed rims looked as though they had shed tears of blood . He excited feelings of repulsion in some , and of pity in others . The young medical students who came to the house noticed the drooping of his lower lip and the conformation of the facial angle ; and , after teasing him for some time to no purpose , they declared that cretinism was setting in . One evening after dinner Mme . Vauquer said half banter
 ingly to him , " So those daughters of yours do n't come to see you any more , eh ? " meaning to imply her doubts as to his paternity ; but Father Goriot shrank as if his hostess had touched him with a sword-point . " They come sometimes , " he said in a tremulous voice . " Aha ! you still see them sometimes ? " cried the students . " Bravo , Father Goriot ! " The old man scarcely seemed to hear the witticisms at his expense that followed on the words ; he had relapsed into the dreamy state of mind that these superficial observers took for senile torpor , due to his lack of intelligence . If they had only known , they might have been deeply interested by the problem of his condition ; but few problems were more obscure . It was easy , of course , to find out whether Goriot had really been a vermicelli manufacturer ; the amount of his fortune was readily discoverable ; but the old people , who were most inquisitive as to his concerns , never went beyond the limits of the Quarter , an
 d lived in the lodging-house much as oysters cling to a rock . As for the rest , the current of life in Paris daily awaited them , and swept them away with it ; so soon as they left the Rue Neuve-Sainte-Genevieve , they forgot the existence of the old man , their butt at dinner . For those narrow souls , or for careless youth , the misery in Father Goriot 's withered face and its dull apathy were quite incompatible with wealth or any sort of intelligence . As for the creatures whom he called his daughters , all Mme . Vauquer 's boarders were of her opinion . With the faculty for severe logic sedulously cultivated by elderly women during long evenings of gossip till they can always find an hypothesis to fit all circumstances , she was wont to reason thus : " If Father Goriot had daughters of his own as rich as those ladies who came here seemed to be , he would not be lodging in my house , on the third floor , at forty-five francs a month ; and he would not go about dressed like a poo
 r man . " No objection could be raised to these inferences . So by the end of the month of November 1819 , at the time when the curtain rises on this drama , every one in the house had come to have a very decided opinion as to the poor old man . He had never had either wife or daughter ; excesses had reduced him to this sluggish condition ; he was a sort of human mollusk who should be classed among the capulidoe , so one of the dinner contingent , an _employe_ at the Museum , who had a pretty wit of his own . Poiret was an eagle , a gentleman , compared with Goriot . Poiret would join the talk , argue , answer when he was spoken to ; as a matter of fact , his talk , arguments , and responses contributed nothing to the conversation , for Poiret had a habit of repeating what the others said in different words ; still , he did join in the talk ; he was alive , and seemed capable of feeling ; while Father Goriot ( to quote the Museum official again ) was invariably at zero degrees--Reau
 mur . Eugene de Rastignac had just returned to Paris in a state of mind not unknown to young men who are conscious of unusual powers , and to those whose faculties are so stimulated by a difficult position , that for the time being they rise above the ordinary level . Rastignac 's first year of study for the preliminary examinations in law had left him free to see the sights of Paris and to enjoy some of its amusements . A student has not much time on his hands if he sets himself to learn the repertory of every theatre , and to study the ins and outs of the labyrinth of Paris . To know its customs ; to learn the language , and become familiar with the amusements of the capital , he must explore its recesses , good and bad , follow the studies that please him best , and form some idea of the treasures contained in galleries and museums . At this stage of his career a student grows eager and excited about all sorts of follies that seem to him to be of immense importance . He has his h
 ero , his great man , a professor at the College de France , paid to talk down to the level of his audience . He adjusts his cravat , and strikes various attitudes for the benefit of the women in the first galleries at the Opera-Comique . As he passes through all these successive initiations , and breaks out of his sheath , the horizons of life widen around him , and at length he grasps the plan of society with the different human strata of which it is composed . If he begins by admiring the procession of carriages on sunny afternoons in the Champs-Elysees , he soon reaches the further stage of envying their owners . Unconsciously , Eugene had served his apprenticeship before he went back to Angouleme for the long vacation after taking his degrees as bachelor of arts and bachelor of law . The illusions of childhood had vanished , so also had the ideas he brought with him from the provinces ; he had returned thither with an intelligence developed , with loftier ambitions , and saw th
 ings as they were at home in the old manor house . His father and mother , his two brothers and two sisters , with an aged aunt , whose whole fortune consisted in annuities , lived on the little estate of Rastignac . The whole property brought in about three thousand francs ; and though the amount varied with the season ( as must always be the case in a vine-growing district ) , they were obliged to spare an unvarying twelve hundred francs out of their income for him . He saw how constantly the poverty , which they had generously hidden from him , weighed upon them ; he could not help comparing the sisters , who had seemed so beautiful to his boyish eyes , with women in Paris , who had realized the beauty of his dreams . The uncertain future of the whole family depended upon him . It did not escape his eyes that not a crumb was wasted in the house , nor that the wine they drank was made from the second pressing ; a multitude of small things , which it is useless to speak of in detai
 l here , made him burn to distinguish himself , and his ambition to succeed increased tenfold . He meant , like all great souls , that his success should be owing entirely to his merits ; but his was pre-eminently a southern temperament , the execution of his plans was sure to be marred by the vertigo that seizes on youth when youth sees itself alone in a wide sea , uncertain how to spend its energies , whither to steer its course , how to adapt its sails to the winds . At first he determined to fling himself heart and soul into his work , but he was diverted from this purpose by the need of society and connections ; then he saw how great an influence women exert in social life , and suddenly made up his mind to go out into this world to seek a protectress there . Surely a clever and high-spirited young man , whose wit and courage were set off to advantage by a graceful figure and the vigorous kind of beauty that readily strikes a woman 's imagination , need not despair of finding a
  protectress . These ideas occurred to him in his country walks with his sisters , whom he had once joined so gaily . The girls thought him very much changed . His aunt , Mme . de Marcillac , had been presented at court , and had moved among the brightest heights of that lofty region . Suddenly the young man 's ambition discerned in those recollections of hers , which had been like nursery fairy tales to her nephews and nieces , the elements of a social success at least as important as the success which he had achieved at the Ecole de Droit . He began to ask his aunt about those relations ; some of the old ties might still hold good . After much shaking of the branches of the family tree , the old lady came to the conclusion that of all persons who could be useful to her nephew among the selfish genus of rich relations , the Vicomtesse de Beauseant was the least likely to refuse . To this lady , therefore , she wrote in the old-fashioned style , recommending Eugene to her ; pointing
  out to her nephew that if he succeeded in pleasing Mme . de Beauseant , the Vicomtesse would introduce him to other relations . A few days after his return to Paris , therefore , Rastignac sent his aunt 's letter to Mme . de Beauseant . The Vicomtesse replied by an invitation to a ball for the following evening . This was the position of affairs at the Maison Vauquer at the end of November 1819. A few days later , after Mme . de Beauseant 's ball , Eugene came in at two o'clock in the morning . The persevering student meant to make up for the lost time by working until daylight . It was the first time that he had attempted to spend the night in this way in that silent quarter . The spell of a factitious energy was upon him ; he had beheld the pomp and splendor of the world . He had not dined at the Maison Vauquer ; the boarders probably would think that he would walk home at daybreak from the dance , as he had done sometimes on former occasions , after a fete at the Prado , or a ba
 ll at the Odeon , splashing his silk stockings thereby , and ruining his pumps . It so happened that Christophe took a look into the street before drawing the bolts of the door ; and Rastignac , coming in at that moment , could go up to his room without making any noise , followed by Christophe , who made a great deal . Eugene exchanged his dress suit for a shabby overcoat and slippers , kindled a fire with some blocks of patent fuel , and prepared for his night 's work in such a sort that the faint sounds he made were drowned by Christophe 's heavy tramp on the stairs . Eugene sat absorbed in thought for a few moments before plunging into his law books . He had just become aware of the fact that the Vicomtesse de Beauseant was one of the queens of fashion , that her house was thought to be the pleasantest in the Faubourg Saint-Germain . And not only so , she was , by right of her fortune , and the name she bore , one of the most conspicuous figures in that aristocratic world . Than
 ks to the aunt , thanks to Mme . de Marcillac 's letter of introduction , the poor student had been kindly received in that house before he knew the extent of the favor thus shown to him . It was almost like a patent of nobility to be admitted to those gilded salons ; he had appeared in the most exclusive circle in Paris , and now all doors were open for him . Eugene had been dazzled at first by the brilliant assembly , and had scarcely exchanged a few words with the Vicomtesse ; he had been content to single out a goddess among this throng of Parisian divinities , one of those women who are sure to attract a young man 's fancy . The Comtesse Anastasie de Restaud was tall and gracefully made ; she had one of the prettiest figures in Paris . Imagine a pair of great dark eyes , a magnificently moulded hand , a shapely foot . There was a fiery energy in her movements ; the Marquis de Ronquerolles had called her " a thoroughbred , " " a pure pedigree , " these figures of speech have rep
 laced the " heavenly angel " and Ossianic nomenclature ; the old mythology of love is extinct , doomed to perish by modern dandyism . But for Rastignac , Mme . Anastasie de Restaud was the woman for whom he had sighed . He had contrived to write his name twice upon the list of partners upon her fan , and had snatched a few words with her during the first quadrille . " Where shall I meet you again , Madame ? " he asked abruptly , and the tones of his voice were full of the vehement energy that women like so well . " Oh , everywhere ! " said she , " in the Bois , at the Bouffons , in my own house . " With the impetuosity of his adventurous southern temper , he did all he could to cultivate an acquaintance with this lovely countess , making the best of his opportunities in the quadrille and during a waltz that she gave him . When he told her that he was a cousin of Mme . de Beauseant 's , the Countess , whom he took for a great lady , asked him to call at her house , and after her part
 ing smile , Rastignac felt convinced that he must make this visit . He was so lucky as to light upon some one who did not laugh at his ignorance , a fatal defect among the gilded and insolent youth of that period ; the coterie of Maulincourts , Maximes de Trailles , de Marsays , Ronquerolles , Ajuda-Pintos , and Vandenesses who shone there in all the glory of coxcombry among the best-dressed women of fashion in Paris--Lady Brandon , the Duchesse de Langeais , the Comtesse de Kergarouet , Mme . de Serizy , the Duchesse de Carigliano , the Comtesse Ferraud , Mme . de Lanty , the Marquise d'Aiglemont , Mme . Firmiani , the Marquise de Listomere and the Marquise d'Espard , the Duchesse de Maufrigneuse and the Grandlieus . Luckily , therefore , for him , the novice happened upon the Marquis de Montriveau , the lover of the Duchesse de Langeais , a general as simple as a child ; from him Rastignac learned that the Comtesse lived in the Rue du Helder . Ah , what it is to be young , eager t
 o see the world , greedily on the watch for any chance that brings you nearer the woman of your dreams , and behold two houses open their doors to you ! To set foot in the Vicomtesse de Beauseant 's house in the Faubourg Saint-Germain ; to fall on your knees before a Comtesse de Restaud in the Chaussee d'Antin ; to look at one glance across a vista of Paris drawing-rooms , conscious that , possessing sufficient good looks , you may hope to find aid and protection there in a feminine heart ! To feel ambitious enough to spurn the tight-rope on which you must walk with the steady head of an acrobat for whom a fall is impossible , and to find in a charming woman the best of all balancing poles . He sat there with his thoughts for a while , Law on the one hand , and Poverty on the other , beholding a radiant vision of a woman rise above the dull , smouldering fire . Who would not have paused and questioned the future as Eugene was doing ? who would not have pictured it full of success ? 
 His wondering thoughts took wings ; he was transported out of the present into that blissful future ; he was sitting by Mme . de Restaud 's side , when a sort of sigh , like the grunt of an overburdened St. Joseph , broke the silence of the night . It vibrated through the student , who took the sound for a death groan . He opened his door noiselessly , went out upon the landing , and saw a thin streak of light under Father Goriot 's door . Eugene feared that his neighbor had been taken ill ; he went over and looked through the keyhole ; the old man was busily engaged in an occupation so singular and so suspicious that Rastignac thought he was only doing a piece of necessary service to society to watch the self-styled vermicelli maker 's nocturnal industries . The table was upturned , and Goriot had doubtless in some way secured a silver plate and cup to the bar before knotting a thick rope round them ; he was pulling at this rope with such enormous force that they were being crushed
  and twisted out of shape ; to all appearance he meant to convert the richly wrought metal into ingots . " _Peste ! _ what a man ! " said Rastignac , as he watched Goriot 's muscular arms ; there was not a sound in the room while the old man , with the aid of the rope , was kneading the silver like dough . " Was he then , indeed , a thief , or a receiver of stolen goods , who affected imbecility and decrepitude , and lived like a beggar that he might carry on his pursuits the more securely ? " Eugene stood for a moment revolving these questions , then he looked again through the keyhole . Father Goriot had unwound his coil of rope ; he had covered the table with a blanket , and was now employed in rolling the flattened mass of silver into a bar , an operation which he performed with marvelous dexterity . " Why , he must be as strong as Augustus , King of Poland ! " said Eugene to himself when the bar was nearly finished . Father Goriot looked sadly at his handiwork , tears fell from
  his eyes , he blew out the dip which had served him for a light while he manipulated the silver , and Eugene heard him sigh as he lay down again . " He is mad , " thought the student . " _Poor child ! _ " Father Goriot said aloud . Rastignac , hearing those words , concluded to keep silence ; he would not hastily condemn his neighbor . He was just in the doorway of his room when a strange sound from the staircase below reached his ears ; it might have been made by two men coming up in list slippers . Eugene listened ; two men there certainly were , he could hear their breathing . Yet there had been no sound of opening the street door , no footsteps in the passage . Suddenly , too , he saw a faint gleam of light on the second story ; it came from M. Vautrin 's room . " There are a good many mysteries here for a lodging-house ! " he said to himself . He went part of the way downstairs and listened again . The rattle of gold reached his ears . In another moment the light was put out ,
  and again he distinctly heard the breathing of two men , but no sound of a door being opened or shut . The two men went downstairs , the faint sounds growing fainter as they went . " Who is there ? " cried Mme . Vauquer out of her bedroom window . " I , Mme . Vauquer , " answered Vautrin 's deep bass voice . " I am coming in . " " That is odd ! Christophe drew the bolts , " said Eugene , going back to his room . " You have to sit up at night , it seems , if you really mean to know all that is going on about you in Paris . " These incidents turned his thought from his ambitious dreams ; he betook himself to his work , but his thought wandered back to Father Goriot 's suspicious occupation ; Mme . de Restaud 's face swam again and again before his eyes like a vision of a brilliant future ; and at last he lay down and slept with clenched fists . When a young man makes up his mind that he will work all night , the chances are that seven times out of ten he will sleep till morning . Suc
 h vigils do not begin before we are turned twenty . The next morning Paris was wrapped in one of the dense fogs that throw the most punctual people out in their calculations as to the time ; even the most business-like folk fail to keep their appointments in such weather , and ordinary mortals wake up at noon and fancy it is eight o'clock . On this morning it was half-past nine , and Mme . Vauquer still lay abed . Christophe was late , Sylvie was late , but the two sat comfortably taking their coffee as usual . It was Sylvie 's custom to take the cream off the milk destined for the boarders ' breakfast for her own , and to boil the remainder for some time , so that madame should not discover this illegal exaction . " Sylvie , " said Christophe , as he dipped a piece of toast into the coffee , " M. Vautrin , who is not such a bad sort , all the same , had two people come to see him again last night . If madame says anything , mind you say nothing about it . " " Has he given you somet
 hing ? " " He gave me a five-franc piece this month , which is as good as saying , 'Hold your tongue . ' " " Except him and Mme . Couture , who does n't look twice at every penny , there 's no one in the house that does n't try to get back with the left hand all that they give with the right at New Year , " said Sylvie . " And , after all , " said Christophe , " what do they give you ? A miserable five-franc piece . There is Father Goriot , who has cleaned his shoes himself these two years past . There is that old beggar Poiret , who goes without blacking altogether ; he would sooner drink it than put it on his boots . Then there is that whipper-snapper of a student , who gives me a couple of francs . Two francs will not pay for my brushes , and he sells his old clothes , and gets more for them than they are worth . Oh ! they 're a shabby lot ! " " Pooh ! " said Sylvie , sipping her coffee , " our places are the best in the Quarter , that I know . But about that great big chap Vautr
 in , Christophe ; has any one told you anything about him ? " " Yes . I met a gentleman in the street a few days ago ; he said to me , 'There 's a gentleman in your place , is n't there ? a tall man that dyes his whiskers ? ' I told him , 'No , sir ; they are n't dyed . A gay fellow like him has n't the time to do it . ' And when I told M. Vautrin about it afterwards , he said , 'Quite right , my boy . That is the way to answer them . There is nothing more unpleasant than to have your little weaknesses known ; it might spoil many a match . ' " " Well , and for my part , " said Sylvie , " a man tried to humbug me at the market wanting to know if I had seen him put on his shirt . Such bosh ! There , " she cried , interrupting herself , " that 's a quarter to ten striking at the Val-de-Grace , and not a soul stirring ! " " Pooh ! they are all gone out . Mme . Couture and the girl went out at eight o'clock to take the wafer at Saint-Etienne . Father Goriot started off somewhere with a p
 arcel , and the student wo n't be back from his lecture till ten o'clock . I saw them go while I was sweeping the stairs ; Father Goriot knocked up against me , and his parcel was as hard as iron . What is the old fellow up to , I wonder ? He is as good as a plaything for the rest of them ; they can never let him alone ; but he is a good man , all the same , and worth more than all of them put together . He does n't give you much himself , but he sometimes sends you with a message to ladies who fork out famous tips ; they are dressed grandly , too . " " His daughters , as he calls them , eh ? There are a dozen of them . " " I have never been to more than two--the two who came here . " " There is madame moving overhead ; I shall have to go , or she will raise a fine racket . Just keep an eye on the milk , Christophe ; do n't let the cat get at it . " Sylvie went up to her mistress ' room . " Sylvie ! How is this ? It 's nearly ten o'clock , and you let me sleep like a dormouse ! Such
  a thing has never happened before . " " It 's the fog ; it is that thick , you could cut it with a knife . " " But how about breakfast ? " " Bah ! the boarders are possessed , I 'm sure . They all cleared out before there was a wink of daylight . " " Do speak properly , Sylvie , " Mme . Vauquer retorted ; " say a blink of daylight . " " Ah , well , madame , whichever you please . Anyhow , you can have breakfast at ten o'clock . La Michonnette and Poiret have neither of them stirred . There are only those two upstairs , and they are sleeping like the logs they are . " " But , Sylvie , you put their names together as if---- " " As if what ? " said Sylvie , bursting into a guffaw . " The two of them make a pair . " " It is a strange thing , is n't it , Sylvie , how M. Vautrin got in last night after Christophe had bolted the door ? " " Not at all , madame . Christophe heard M. Vautrin , and went down and undid the door . And here are you imagining that---- ? " " Give me my bodice , an
 d be quick and get breakfast ready . Dish up the rest of the mutton with the potatoes , and you can put the stewed pears on the table , those at five a penny . " A few moments later Mme . Vauquer came down , just in time to see the cat knock down a plate that covered a bowl of milk , and begin to lap in all haste . " Mistigris ! " she cried . The cat fled , but promptly returned to rub against her ankles . " Oh ! yes , you can wheedle , you old hypocrite ! " she said . " Sylvie ! Sylvie ! " " Yes , madame ; what is it ? " " Just see what the cat has done ! " " It is all that stupid Christophe 's fault . I told him to stop and lay the table . What has become of him ? Do n't you worry , madame ; Father Goriot shall have it . I will fill it up with water , and he wo n't know the difference ; he never notices anything , not even what he eats . " " I wonder where the old heathen can have gone ? " said Mme . Vauquer , setting the plates round the table . " Who knows ? He is up to all sort
 s of tricks . " " I have overslept myself , " said Mme . Vauquer . " But madame looks as fresh as a rose , all the same . " The door bell rang at that moment , and Vautrin came through the sitting-room , singing loudly : " 'Tis the same old story everywhere , A roving heart and a roving glance. . " Oh ! Mamma Vauquer ! good-morning ! " he cried at the sight of his hostess , and he put his arm gaily round her waist . " There ! have done---- " " 'Impertinence ! ' Say it ! " he answered . " Come , say it ! Now , is n't that what you really mean ? Stop a bit , I will help you to set the table . Ah ! I am a nice man , am I not ? " For the locks of brown and the golden hair A sighing lover ... " Oh ! I have just seen something so funny---- ... . led by chance . " " What ? " asked the widow . " Father Goriot in the goldsmith 's shop in the Rue Dauphine at half-past eight this morning . They buy old spoons and forks and gold lace there , and Goriot sold a piece of silver plate for a good ro
 und sum . It had been twisted out of shape very neatly for a man that 's not used to the trade . " " Really ? You do n't say so ? " " Yes . One of my friends is expatriating himself ; I had been to see him off on board the Royal Mail steamer , and was coming back here . I waited after that to see what Father Goriot would do ; it is a comical affair . He came back to this quarter of the world , to the Rue des Gres , and went into a money-lender 's house ; everybody knows him , Gobseck , a stuck-up rascal , that would make dominoes out of his father 's bones , a Turk , a heathen , an old Jew , a Greek ; it would be a difficult matter to rob _him_ , for he puts all his coin into the Bank . " " Then what was Father Goriot doing there ? " " Doing ? " said Vautrin . " Nothing ; he was bent on his own undoing . He is a simpleton , stupid enough to ruin himself by running after---- " " There he is ! " cried Sylvie . " Christophe , " cried Father Goriot 's voice , " come upstairs with me . "
  Christophe went up , and shortly afterwards came down again . " Where are you going ? " Mme . Vauquer asked of her servant . " Out on an errand for M. Goriot . " " What may that be ? " said Vautrin , pouncing on a letter in Christophe 's hand . " _Mme . la Comtesse Anastasie de Restaud_ , " he read . " Where are you going with it ? " he added , as he gave the letter back to Christophe . " To the Rue du Helder . I have orders to give this into her hands myself . " " What is there inside it ? " said Vautrin , holding the letter up to the light . " A banknote ? No. " He peered into the envelope . " A receipted account ! " he cried . " My word ! 'tis a gallant old dotard . Off with you , old chap , " he said , bringing down a hand on Christophe 's head , and spinning the man round like a thimble ; " you will have a famous tip . " By this time the table was set . Sylvie was boiling the milk , Mme . Vauquer was lighting a fire in the stove with some assistance from Vautrin , who kept hum
 ming to himself : " The same old story everywhere , A roving heart and a roving glance . " When everything was ready , Mme . Couture and Mlle . Taillefer came in . " Where have you been this morning , fair lady ? " said Mme . Vauquer , turning to Mme . Couture . " We have just been to say our prayers at Saint-Etienne du Mont . To-day is the day when we must go to see M. Taillefer . Poor little thing ! She is trembling like a leaf , " Mme . Couture went on , as she seated herself before the fire and held the steaming soles of her boots to the blaze . " Warm yourself , Victorine , " said Mme . Vauquer . " It is quite right and proper , mademoiselle , to pray to Heaven to soften your father 's heart , " said Vautrin , as he drew a chair nearer to the orphan girl ; " but that is not enough . What you want is a friend who will give the monster a piece of his mind ; a barbarian that has three millions ( so they say ) , and will not give you a dowry ; and a pretty girl needs a dowry nowada
 ys . " " Poor child ! " said Mme . Vauquer . " Never mind , my pet , your wretch of a father is going just the way to bring

<TRUNCATED>

[04/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/181un_CCW_APII_CONF4_SR2.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/181un_CCW_APII_CONF4_SR2.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/181un_CCW_APII_CONF4_SR2.txt
new file mode 100644
index 0000000..c32a27a
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/181un_CCW_APII_CONF4_SR2.txt
@@ -0,0 +1,2 @@
+
+Geneva , 11 December 2002 SUMMARY RECORD OF THE 2nd MEETING Held at the Palais des Nations , Geneva , on Wednesday , 11 December 2002 , at 4.25 p. m. President : Mr. FAESSLER ( Switzerland ) GENERAL EXCHANGE OF VIEWS ( PLENARY ) ( continued ) The meeting was called to order at 4.25 p. m. GENERAL EXCHANGE OF VIEWS ( PLENARY ) ( agenda item 8 ) ( continued ) Mr. DO\u011eAN ( Turkey ) , reiterating his country 's commitment to the elimination of all anti-personnel mines , said that Turkey welcomed any measures adopted to that end , whether at the bilateral or multilateral level . In January 2002 , it had decided to extend indefinitely its moratorium on the export and transfer of anti-personnel landmines , thereby demonstrating its genuine wish to become party to the Ottawa Convention . In addition to concluding bilateral agreements with its neighbouring countries to demine frontier areas , it had also contributed $50,000 to the demining activities conducted in Bosnia and Herzegovina with t
 he assistance of Turkish troops and had provided financial support for demining projects conducted on the northern frontier of Albania , following the Kosovo conflict . By August 2002 , Turkey had already cleared more than 10,000 mines . In addition , it had set up demining and coordination centres and had trained demining teams . Turkey had suffered a great deal from terrorism and from the extensive use by terrorists of anti-personnel and anti-vehicle mines . Regrettably , terrorists did not consider themselves bound by existing legal instruments and attached scant importance to the increasing support enjoyed by such instruments . The international community should fight terrorism in a concerted manner and should use every possible means to bring terrorists and their accomplices to justice . Mr. SEETHARAM ( India ) said that he was obliged to refute the allegations made against his country by the organization Human Rights Watch . India strictly respected Amended Protocol II . Furth
 ermore , the casualty figures cited by Human Rights Watch were unfounded and had clearly not been verified with the authorities , an omission which reflected badly on the organization . CONSIDERATION OF THE DEVELOPMENT OF TECHNOLOGIES TO PROTECT CIVILIANS AGAINST THE INDISCRIMINATE EFFECTS OF MINES ( agenda item 11 ) The PRESIDENT said that no delegation wished to take the floor on the agenda item . OTHER MATTERS ( agenda item 12 ) The PRESIDENT said that no delegation wished to take the floor on the agenda item . CONSIDERATION AND ADOPTION OF THE FINAL DOCUMENTS ( agenda item 13 ) ( CCW/AP . II/CONF.4/CRP.1 , English only ) The PRESIDENT drew attention to the draft report of the Fourth Annual Conference of the States Parties to Amended Protocol II , issued under symbol CCW/AP . II/CONF.4/CRP.1 , in English only , and invited the Conference to take it up paragraph by paragraph before adopting it in its entirety . Paragraphs 1-6 Paragraphs 1-6 were adopted . Paragraph 7 The PRESIDENT
  said that Morocco should be added to the list of States which had notified the depository of their consent to be bound by Amended Protocol II and had participated in the work of the Conference , bringing the number of such States to 46. Paragraph 7 , as completed , was adopted . Paragraph 8 Paragraph 8 was adopted . Paragraph 9 The PRESIDENT said that the secretariat had informed him that the Islamic Republic of Iran should be removed from the list of States not parties to Amended Protocol II which had participated as observers . Mr. HORUMBA ( Observer from Romania ) said that his country should be added to that list . Paragraph 9 , as amended and completed , was adopted . Paragraph 10 Paragraph 10 was adopted . Paragraph 11 Ms. WALKER ( International Campaign to Ban Landmines ) requested that the names of all the member organizations of the International Campaign to Ban Landmines which had attended the public meetings of the Conference should be listed in the paragraph . Paragraph
  11 , as completed , was adopted . Paragraph 12 Increasing the number of vice-presidents of the Annual Conference Mr. WENSLEY ( South Africa ) suggested that the number of vice-presidents of the Annual Conference should be increased to three , to ensure the representation of each group of countries and China in the General Committee of the Conference , which would therefore also be more balanced . That would also entail amending rules 3 and 7 of the rules of procedure . The PRESIDENT said he took it that the Conference wished to increase the number of vice-presidents to three . It was so decided . Mr. WENSLEY ( South Africa ) suggested that the following text should therefore be added to paragraph 12 of the draft report under consideration : \u201c The Conference decided to amend rules 3 and 7 of the rules of procedure in order to increase the number of vice-presidents from two to three to secure a balanced representation of geographical groups in the General Committee of the Conferenc
 e . Amended rule 3 will read as follows : ` The Conference shall elect from among the States parties participating in the Conference a president and three vice-presidents . These officers shall be elected so as to ensure the representative character of the General Committee provided for in rule 7. ' Amended rule 7 will read as follows : ` The General Committee shall be composed of the President , who shall preside , three vice-presidents and the chairs of any other subsidiary organs . ' \u201d Paragraph 12 , as completed , was adopted . Paragraphs 13 and 14 Paragraphs 13 and 14 were adopted . New paragraph 15 The PRESIDENT said that a new paragraph 15 should be added , to read as follows : \u201c At the same meeting , the Conference heard a message from the Secretary-General of the United Nations , which was read out by Mr. Enrique Rom�n-Moray , Deputy Secretary-General of the Conference on Disarmament and Director of the Geneva Branch of the Department for Disarmament Affairs . \u201d New 
 paragraph 15 was adopted . New paragraph 16 ( old paragraph 15 ) New paragraph 16 was adopted . New paragraph 17 ( old paragraph 16 ) The PRESIDENT said that the secretariat had informed him that Croatia and the United States of America should be added to the list of States which had submitted their annual reports . Mr. WENSLEY ( South Africa ) said that his country should also be added to the list . New paragraph 17 , as completed , was adopted . New paragraph 18 ( old paragraph 17 ) Mr. DAHINDEN ( Switzerland ) said that the following text should be added to new paragraph 18 : \u201c Switzerland suggested that a cover page should be added to the national annual reports , to simplify the transmission of information . This suggestion should be taken up in a decision to be adopted at the next Annual Conference ( see annex I ) . \u201d The thinking behind the suggestion was that , where the information was unchanged from one year to the next , there would be no need to submit a full new rep
 ort : instead a mention on the cover page would suffice . New paragraph 18 , as completed , was adopted . New paragraph 19 ( old paragraph 18 ) Draft appeal issued by States parties to Amended Protocol II on the occasion of their Fourth Annual Conference ( CCW/AP . II/CONF.4/CRP.2 ) The PRESIDENT invited the Conference to consider and adopt the draft appeal by the States parties on the occasion of their Fourth Annual Conference , contained in document CCW/AP . II/CONF.4/CRP.2 and referred to in new paragraph 19 , before proceeding to the adoption of the paragraph . Recalling that the draft had been inspired by the appeal submitted by the States parties at their Third Annual Conference , held immediately prior to the Second Review Conference , he said that , in the informal consultations which he had held before the current meeting , one delegation had suggested that the text should be updated by deleting the paragraph which read : \u201c [ We ] welcome the opportunity provided by the S
 econd Review Conference of the States parties to the Convention on Conventional Weapons to examine the operation and status of Amended Protocol II . \u201d The draft appeal issued under symbol CCW/AP . II/CONF.4/CRP.2 , as amended , was adopted . New paragraph 19 was adopted . New paragraph 20 ( old paragraph 19 ) New paragraph 20 was adopted with a minor editorial change to the English text . New paragraph 21 ( old paragraph 20 ) Designation of the president and vice-presidents of the Fifth Annual Conference Mr. HOLZENBERGER ( Germany ) said that several delegations , including his own , had put forward the idea of designating the president and the vice-presidents of the Fifth Annual Conference at the end of the Fourth Conference , so that the incoming president and vice-presidents could themselves take charge of the preparations for the Conference and would then be able to conduct their work on the basis of those preparations . That approach seemed more rational than requesting the o
 utgoing president and vice-presidents to carry out the preparatory work and more conducive to the smooth running of the Conference . Mr. PET�CZ ( Slovakia ) , speaking on behalf of the group of Eastern European States , said that his group supported the idea . Mr. DOROSIN ( United States of America ) wondered whether it would be valid to designate the president and the vice-presidents of the Fifth Annual Conference at the current Conference . The PRESIDENT explained that the designation at the current stage of the president and vice-presidents of the Fifth Annual Conference would have to be confirmed by their election at that Conference . He took it that the Conference wished to designate the president and the vice-presidents of the Fifth Annual Conference at the current stage . It was so decided . The PRESIDENT invited groups of States to put forward candidates for the posts of president and vice-presidents of the Fifth Annual Conference . Mr. PET�CZ ( Slovakia ) , speaking on be
 half of the group of Eastern European States , said that the group proposed Ambassador Tzantchev of Bulgaria as President of the Fifth Annual Conference . Mr. WENSLEY ( South Africa ) , speaking on behalf of the group of non-aligned and other countries , said that the group proposed the representative of South Africa as Vice-President of the Fifth Conference . It was not possible , at the current stage , to give the name of the South African representative who would perform those functions since the tour of duty of the ambassador currently serving in Geneva would end during the coming few months . Mr. FU Zhigang ( China ) said that China proposed that one of its representatives should also be designated Vice-President of the Fifth Annual Conference . Mr. HOLZENBERGER ( Germany ) , speaking on behalf of the group of Western countries , said that the group also proposed the Swiss Ambassador , Mr. Faessler as Vice-President of the Fifth Annual Conference . The PRESIDENT said that , if 
 he heard no objection , he would take it that the Conference wished to designate the Bulgarian Ambassador , Mr. Tzantchev , as President of the Fifth Annual Conference and the representatives of South Africa and China , as well as the Ambassador of Switzerland , Mr. Faessler , as Vice-Presidents of the Conference . It was so decided . The PRESIDENT suggested that new paragraph 21 should be amended and completed in the light of the decision which the Conference had just adopted . It would therefore read as follows : \u201c The Conference decided that the incoming president and vice-presidents would be designated at the end of the current Conference , so as to ensure the continuity of the preparatory work of the presidency . Accordingly , the Conference decided to designate the Bulgarian Ambassador , Mr. Tzantchev , as President of the Annual Conference of States Parties to be held in 2003 and the representatives of South Africa and China , and the Ambassador of Switzerland , Mr. Faessle
 r , as Vice-Presidents . \u201d New paragraph 21 , as modified and completed , was adopted . New paragraph 22 ( old paragraph 21 ) The PRESIDENT drew attention to the estimated costs which would be contained in document CCW/AP . II/CONF.4/3 and which were referred to in new paragraph 22. He pointed out that , in effect , there had been two sets of estimates , depending on whether the Conference lasted one or two days. He explained that the dates given in the estimates were very tentative and recalled that the dates and duration of the Fifth Annual Conference would be decided , as indicated in the paragraph under consideration , at the Meeting of States Parties to the Convention to be held on 12 and 13 December 2002. New paragraph 22 was adopted with a minor editorial change to the English text . New paragraph 23 ( old paragraph 22 ) Mr. FU Zhigang ( China ) questioned the wording of the paragraph , which stated that the President was requested to present the report of the Fourth Annual
  Conference of the States Parties to Amended Protocol II to the Meeting of States Parties to the Convention , to be held on 12 and 13 December 2002 , and wondered whether the States parties to Amended Protocol II were not themselves empowered to adopt the report of their own Fourth Annual Conference and why the report needed to be endorsed by the Meeting of States Parties . It had been his understanding that the Meeting of States Parties would have the sole purpose of considering the procedural report of the Group of Governmental Experts and deciding on follow-up to its work , as agreed by the Second Review Conference of the Convention . Mr. WENSLEY ( South Africa ) pointed out that , in 2001 , the States parties to Amended Protocol II had concluded that they should submit the report of their Third Annual Conference to the forthcoming Second Conference of States Parties , which would be responsible for reviewing the status and the operation of the Convention and its annexed protocol
 s as a whole , including Amended Protocol II . He agreed with the representative of China that the mandate of the Meeting of States Parties on 12 and 13 December was quite different , since it had been tasked with considering the report of the Group of Governmental Experts and deciding on follow-up to its work . While he had no objection in principle to the President of the Fourth Annual Conference reporting to that Meeting , he did not believe that there was any real necessity for such a report . Mr. DA SILVA ( Brazil ) shared that view : the Meeting of States Parties on 12 and 13 December had been called with a view to considering the procedural report of the Group of Governmental Experts and it would be as well to confine it to that task . The PRESIDENT said he took it that all delegations agreed on that point . Accordingly , the second part of new paragraph 23 should be deleted and the paragraph should read as follows : \u201c At its final meeting , on 11 December 2002 , the Fourth
  Annual Conference adopted its report . \u201d New paragraph 23 , as amended , was adopted . The draft report of the Fourth Annual Conference of the States Parties to Amended Protocol II was adopted in its entirety . The PRESIDENT said that the Conference had concluded the consideration of its agenda . The PRESIDENT thanked all delegations and the staff of the secretariat and of Conference Services for their assistance and declared the Fourth Annual Conference of the States Parties to Amended Protocol II closed . The meeting rose at 5.30 p. m. This record is subject to correction . Corrections should be submitted in one of the working languages . They should be set forth in a memorandum and also incorporated in a copy of the record . They should be sent within one week of the date of this document to the Official Records Editing Section , room E.4108 , Palais des Nations , Geneva . Any corrections to the records of the meetings of this Conference will be consolidated in a single corrig
 endum , to be issued shortly after the end of the Conference . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/182un_CD_1605.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/182un_CD_1605.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/182un_CD_1605.txt
new file mode 100644
index 0000000..6ede6da
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/182un_CD_1605.txt
@@ -0,0 +1,2 @@
+
+27 January 2000 Original : ARABIC , CHINESE , ENGLISH , FRENCH , On 9 and 10 December 1999 , the President of the People 's Republic of China , Mr. Jiang Zemin and the President of the Russian Federation , Mr. Boris Yeltsin , held a second informal summit during which they had an in-depth exchange of views on issues of common interest . The two leaders issued a joint statement on 10 December 1999. Please find attached the text of the aforementioned joint statement in the Chinese and Russian languages . We would be grateful if the joint statement could be issued and ciruclated as an official document of the Conference on Disarmament . ( Signed ) HU Xiaodi ( Signed ) Vasily SIDOROV Ambassador Ambassador Deputy Permanent Representative Permanent Representative Head of delegation of the Head of delegation of the People 's Republic of China to the Russian Federation to the Conference on Disarmament Conference on Disarmament Annex Joint Chinese-Russian statement adopted on 10 December 199
 9 In the course of their second informal meeting , held in Beijing on 9 and 10 December 1999 , Mr. Jiang Zemin , President of the People 's Republic of China , and Mr. B. N. Yeltsin , President of the Russian Federation , held an in-depth exchange of views on questions of mutual interest . Afterwards , the two heads of State adopted the following statement : The two heads of State emphasize the importance and timeliness of the conclusions and assessments contained in the Russian-Chinese Joint Declaration on a Multipolar World and the Establishment of a New International Order of 23 April 1997 , the joint Russian-Chinese statement of 10 November 1997 , the joint statement on the results of the Russian-Chinese summit meeting on Russian-Chinese relations on the threshold of the twenty-first century , and the joint communiqu� on the results of the Russian-Chinese summit meeting of 23 November 1998. The leaders of China and the Russian Federation reaffirm that they will strictly adhere 
 to the principled approaches to international questions set forth in the above-mentioned documents . They advocate the establishment in the twenty-first century of a multipolar world based on the Charter of the United Nations and the applicable norms of international law , the strengthening of the leading role of the United Nations in international affairs , the peaceful settlement of international disputes by political means , equitable relations among all members of the international community and equal security for all , respect for each other 's choice of the path of development and each other 's sovereignty , non-interference in internal affairs and the creation of a just , equitable and mutually advantageous international political and economic order . The two sides appeal to all countries of the world to make concerted efforts for the creation of a democratic and balanced multipolar world structure in which the different cultures would exist harmoniously , cooperate construct
 ively and enrich each other , and in which the equality of all sovereign States would be ensured . The two sides feel obliged to note the recent growth of unfavourable trends in the sphere of international security . The two sides consider that the creation , in violation of the Anti-Ballistic Missile Treaty of 1972 , of a national anti-ballistic-missile defence system by one of the States which signed that Treaty will have a destructive effect on strategic stability and on the entire structure of key international agreements relating to disarmament and the non-proliferation of weapons of mass destruction and means of their delivery , and will mar prospects for achieving further progress in these directions . The two sides reaffirm that they consider unacceptable any attempts to violate the Anti-Ballistic Missile Treaty of 1972 and emphasize that the Treaty has been and must remain a cornerstone of strategic stability . The two sides hold the view that the 1972 Treaty must be strict
 ly and fully implemented . In this connection , the sides note in particular the importance of General Assembly resolution 54/54 A , of 1 December 1999 , entitled \u201c Preservation of and compliance with the Treaty on the Limitation of Anti-Ballistic Missile Systems \u201d , submitted by China , the Russian Federation and Belarus . The two sides point out that the plans of certain Governments to develop a regional anti-ballistic-missile defence system are liable to undermine peace and stability in the Asia and Pacific region . The Russian side supports the position of the Chinese side to the effect that the inclusion of the Chinese province of Taiwan in these plans in any form by any Government whatsoever is unacceptable . The two sides express deep regret over the refusal by the United States of America to ratify the Comprehensive Nuclear-Test-Ban Treaty . The two countries stress that they share identical or similar positions on a number of key issues , and this is a major prerequisit
 e for joint or coordinated efforts to maintain strategic stability . These key issues include the prevention of an arms race in outer space or through the use of information technology , implementation of the treaty on the Non-Proliferation of Nuclear Weapons and the Chemical Weapons Convention , the elaboration of a protocol to the Biological Weapons Convention and the drafting of an agenda for the Conference on Disarmament . The Chinese side expresses its understanding of the position of the Russian side with regard to the further reduction of the strategic offensive weapons of the United States of America and the Russian Federation . The two sides consider that full responsibility for the consequences of any undermining of strategic stability and international security would rest with those States which caused the collapse of fundamental agreements in the field of disarmament . The sides maintain that , in the present situation , there is a growing need for close coordination of 
 actions in the maintenance of global strategic stability . The two sides observe that certain adverse trends are increasingly discernible in international relations . These include the endeavour to foist on the international community a unipolar global structure and a single system of culture , values and ideology ; attempts to weaken the role of the United Nations and its Security Council ; the search for grounds to justify the irresponsible interpretation or revision of the fundamental contents and principles of the Charter of the United Nations ; the strengthening and expansion of military blocs ; the replacement of international law by pressure tactics and even the resort to military force ; the use of the concept which holds that \u201c human rights take precedence over sovereignty \u201d and of the theories of \u201c humanitarian intervention \u201d to infringe the sovereign rights of independent States . The two sides express their readiness , together with other countries , to oppose su
 ch trends , which prevent the establishment of an equitable , multipolar structure of international relationsIV The two sides emphasize that a major priority of the foreign-policy efforts of China and the Russian Federation is the affirmation of the leading role of the United Nations in world affairs . The two sides agree that the rational and carefully administered reform of the Organization will help to strengthen its authority and role . At the same time , the two sides express the conviction that the status and role of the Security Council , as the principal organ responsible for the maintenance of international peace and security , must under no circumstances be placed in doubt or weakened . The sides once again declare that the retention unchanged of the powers vested under the Charter in the existing permanent members of the Security Council is essential for maintaining the effectiveness and stability of the United Nations . China and the Russian Federation attach great impor
 tance to their admission to membership in the World Trade Organization ( WTO ) under conditions that would enable them to occupy an equitable and worthy place in the world-trade system , which would be in keeping with the interests of all members of that Organization . Negotiations on the admission of China and the Russian Federation into the WTO have reached a decisive stage . The two sides consider that the WTO should be universal in character and that , with regard to the conditions for the admission of new members , a balance should be maintained between rights and obligations , with consideration also being given to their level of socio-economic development . The two sides maintain that international terrorism , religious extremism and national separatism in all its forms , as well as transnational criminal activities ( such as illicit trafficking in arms , narcotic drugs and psychotropic and explosive substances , and economic offences ) have now become serious threats to the 
 security of sovereign States and to international peace and security . The two sides reaffirm their determination to take practical , coordinated steps to combat those phenomena on both a bilateral and a multilateral basis . China and the Russian Federation express their satisfaction at the pace of implementation of the Bishkek Declaration , adopted by the heads of States members of the \u201c Shanghai Five \u201d ( the People 's Republic of China , the Republic of Kazakhstan , the Kyrgyz Republic , the Russian Federation and the Republic of Tajikistan ) on 25 August 1999 ( A/54/314-S/1999/942 , annex ) . The meeting of the senior law-enforcement officials of the five States held in Bishkek on 1 and 2 December 1999 was a significant practical step in that direction . The two sides consider that the meeting of ministers of defence of the countries of the \u201c Shanghai Five \u201d to be held in the year 2000 will be an important step towards implementing the agreements reached by the five heads
  of State . The two sides consider that the time is now ripe to begin preparations for a meeting of ministers for foreign affairs of the five States . The sides support the idea of conducting consultations among experts of the Five on questions of multilateral economic cooperation ( including cooperation in the transport field and in the extraction and transport of oil and gas ) . Such talks could become an important and integral part of preparations for a meeting of the heads of State of the People 's Republic of China , the Republic of Kazakhstan , the Kyrgyz Republic , the Russian Federation and the Republic of Tajikistan . The two sides underscore the need for the earliest possible resolution of the question of Iraq exclusively through political means in strict accordance with the norms of international law , in keeping with the central role of the Security Council and on the basis of the resolutions it has adopted . The resumption by Iraq of cooperation with the United Nations 
 must be accompanied by the lifting of the economic sanctions imposed on that country . The two sides expressed serious concern about the development of the situation in and around Kosovo and the escalation of inter-ethnic tension in the region , particularly in connection with the unsatisfactory level of implementation of Security Council resolution 1244 ( 1999 ) and the growing trend towards separating Kosovo from the Federal Republic of Yugoslavia . China and the Russian Federation , as two permanent members of the Security Council , urge all interested parties to ensure the full and unconditional implementation of all provisions of resolution 1244 ( 1999 ) , full respect for the sovereignty and territorial integrity of the Federal Republic of Yugoslavia , the comprehensive and consistent observance of the Charter of the United Nations and of the universally recognized principles of international law , and respect for the legitimate rights of all nationalities in the Kosovo region
  . The two sides reaffirm their mutual support for the maintenance of each other 's national unity , sovereignty and territorial integrity . The Russian side , in supporting the important issue of the reunification of China , reaffirmed its position on the question of Taiwan set forth in earlier documents adopted jointly by the two sides , and declared that it could not accept affirmations that relations between the two sides situated on the opposing shores of the Taiwan Strait were \u201c relations between States \u201d . The Chinese side expressed satisfaction regarding the above-mentioned position of the Russian side . The Chinese side again declared that the Chechen problem was entirely an internal affair of the Russian Federation . The Chinese side supports the actions being taken by the Government of the Russian Federation to combat terrorist and separatist forces . The Russian side expressed satisfaction with regard to the above-mentioned position of the Chinese side . China and th
 e Russian Federation , in pursuance of their goal of achieving an equitable and trusting strategic partnership , and also on the basis of the convergence or similarity of their positions both on questions of principle relating to the building of a multipolar , democratic and equitable world order and on specific current international problems , will strengthen cooperation in those directions . The cooperation of the two sides in international affairs is not directed against third countries ; rather , it is aimed at protecting their own fundamental national interests , but also at strengthening peace and stability in the Asia and Pacific region and throughout the world . Previously issued as document A/54/681-S/1999/1264 dated 20 December 1999. 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/183un_CD_1606.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/183un_CD_1606.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/183un_CD_1606.txt
new file mode 100644
index 0000000..d791271
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/183un_CD_1606.txt
@@ -0,0 +1,2 @@
+
+9 February 2000 Original : CHINESE LETTER DATED 9 FEBRUARY 2000 FROM THE PERMANENT REPRESENTATIVE OF CHINA TO THE CONFERENCE ON DISARMAMENT ADDRESSED TO THE SECRETARY-GENERAL OF THE CONFERENCE TRANSMITTING A WORKING PAPER ENTITLED \u201c CHINA'S POSITION ON AND SUGGESTIONS FOR WAYS TO ADDRESS THE ISSUE OF PREVENTION OF AN ARMS RACE IN OUTER SPACE AT THE CONFERENCE ON DISARMAMENT \u201d On instructions from the Chinese Government , I have the honour to attach herewith a working paper entitled \u201c China 's position on and Suggestions for Ways to Address the Issue of Prevention of an Arms Race in Outer Space at the Conference on Disarmament \u201d . I would be grateful if this document could be issued and circulated as an official document of the Conference on Disarmament . ( Signed ) Hu Xiaodi Ambassador for Disarmament Affairs Head of the Delegation of China to the Conference on Disarmament China Working Paper CHINA'S POSITION ON AND SUGGESTIONS FOR WAYS TO ADDRESS Outer space belongs to all 
 mankind . All countries have equal rights in the exploration and use of outer space for peaceful purposes although their levels of economic and scientific development may differ . It is the shared desire of all mankind to forestall the spread of weapons and an arms race in outer space . Some people believe that since currently there is no arms race in outer space , the CD has no need to discuss its prevention or negotiate the conclusion of international legal instruments in this regard . However , history and reality have both shown not only that there are indeed attempts , programmes and moves unilaterally to seek military and strategic superiority in or control over outer space but that there have been new developments in this respect . Such development , if unchecked , may lead to the weaponization of outer space in the near future or even to a multilateral arms race in outer space . Therefore , it is a present and pressing necessity for the international community to take effect
 ive measures to stop such negative developments . As the single multilateral disarmament negotiating forum , the Conference on Disarmament ( Conference ) should concentrate on the most pressing and prominent issues in international arms control and disarmament , the ones that have the greatest bearing on global peace and security in the twenty-first century . PAROS is one such important issue , and should therefore be a top priority at the Conference . The Conference should play a primary role in the negotiations to prevent any form of arms race in outer space . At its fifty-fourth session the United Nations General Assembly once again adopted , by an overwhelming majority , a resolution on PAROS . It was reaffirmed that negotiating an international agreement or agreements to prevent an arms race in outer space remains a priority task of the CD 's Ad Hoc Committee . The fact that the resolution was adopted without opposition reflects the common aspiration and insistent demand of the
  international community to prevent an arms race in outer space . The General Assembly also adopted at its fifty-fourth session , and also by an overwhelming majority , a resolution on preservation of and compliance with the Anti-Ballistic Missile Treaty , an issue that is related to the prevention of an arms race in outer space . In the resolution the General Assembly recognizes the historical role of the 1972 Treaty as a cornerstone for maintaining global peace and security and strategic stability , reaffirms its continued validity and relevance in the current international situation and supports further efforts by the international community in the light of emerging developments with the goal of safeguarding the inviolability and integrity of the ABM Treaty in which the international community bears strong interest . Since PAROS was put on the CD agenda in 1982 , the Conference has , through the establishment of the Ad Hoc Committee and other means , held discussions on definitio
 ns , principles , existing treaties and confidence-building measures , and accumulated experience in this field , preparing the ground for future work in this area . With the accelerated development of outer space weapons , anti-ballistic missiles and other weapon systems , individual countries have stepped up efforts to secure military superiority in outer space and have mapped out and are pursuing plans to secure military superiority on the ground from space . In these circumstances , preventing outer space from becoming a new venue for an arms race without prejudice to its peaceful uses has obviously become the most important and pressing task of the Conference . To accomplish this , the Conference must first re-establish the Ad Hoc Committee under agenda item 3 to negotiate and conclude an international legal instrument prohibiting the testing , deployment and use of weapons , weapon systems and components in outer space so as to prevent the weaponization of , and an arms race i
 n , outer space . In carrying out its mandate , the Ad Hoc Committee must take into account all relevant developments and specific proposals , present and future . As a preliminary step towards the negotiation of an international legal instrument , it might discuss and review all pertinent issues , including current military activities in outer space and related developments , their influence on the prevention of an arms race in outer space ; shortcomings in the existing international instruments ; and the basic elements of the future international legal instrument . The Chinese delegation has taken note of the various ideas and suggestions on PAROS put forward in the CD . China believes that the re-established Ad Hoc Committee should be an open-ended , all-embracing mechanism where all participants may air and discuss different views . It should set as its ultimate goal and clear mandate the negotiation and conclusion of one or several international legal instruments to prevent the
  weaponization of and an arms race in outer space . A number of international legal instruments on the peaceful uses of outer space and the prevention of an arms race in outer space have been concluded . The 1963 Treaty Banning Nuclear Weapon Tests in the Atmosphere , in Outer Space and Under Water prohibits any nuclear weapon test explosion in outer space . The 1996 Comprehensive Nuclear Test Ban Treaty prohibits any nuclear weapon test explosion in any circumstances . According to the 1967 Treaty on Principles Governing the Activities of States in the Exploration and Use of Outer Space , including the Moon and other Celestial Bodies , outer space , including the moon and other celestial bodies , is not subject to national appropriation by claim of sovereignty , by means of use or occupation , or by any other means . States Parties to the Treaty undertake not to place in orbit around the Earth any objects carrying nuclear weapons or any other kinds of weapons of mass destruction , 
 install such weapons on celestial bodies , or station such weapons in outer space in any other manner . The establishment of military bases , installations and fortifications , the testing of any type of weapons and the conduct of military manoeuvres on celestial bodies is forbidden . The 1972 Treaty between the United States and the Union of Soviet Socialist Republics on the Limitation of Anti-Ballistic Missile Systems prohibits the development , testing and deployment of space-based ABM systems or components . Besides these , the 1972 Convention on International Liability for Damage Caused by Space Objects , the 1975 Convention on Registration of Objects Launched into Outer Space and the 1979 Agreement Governing the Activities of States on the Moon and other Celestial Bodies all contain provisions on outer space activities which have helped to constrain some aspects of military activities in outer space to some extent . However , these instruments have been ineffective in preventi
 ng the weaponization of and an arms race in outer space . Some have imposed limited prohibitions and contained many loopholes and ambiguities . Some have not been fully complied with or are in danger of being violated , amended or even abrogated . Most crucially , as they have failed to reflect the latest developments in aerospace technology they cannot prevent the potential weaponization of outer space or an arms race in outer space in the twenty-first century . The Chinese delegation believes that the most direct and effective way to prevent the weaponization of and an arms race in outer space is to negotiate and conclude new international legal instruments while strictly observing the existing bilateral and multilateral agreements . CHINA'S BASIC POSITION ON PAROS China has always opposed arms races , in outer space and elsewhere . It maintains that the exploration and use of outer space should only serve to promote countries ' economic , scientific and cultural development and b
 enefit all mankind . With the use of military satellites , outer space has already been militarized to some extent . Military satellites involve rather complex issues and their role should not be all together negated . Therefore , the primary goal at present in our efforts to prevent the weaponization of and an arms race in outer space is to ban the testing , deployment and use of weapons , weapon systems and components in outer space . What should be particularly emphasized is that the Powers with the greatest space capabilities bear a special responsibility for preventing the weaponization of and an arms race in outer space and ensuring the use of space for peaceful purposes . Pending the conclusion of a new multilateral legal instrument on the prevention of an arms race in outer space , all countries concerned should undertake not to test , deploy or use any weapons , weapon systems or components in outer space . The Chinese delegation tentatively suggests that the new internatio
 nal legal instruments to prevent the weaponization of and an arms race in outer space , in whatever form or by whatever name , might contain the following basic elements : Purposes : to prevent the weaponization of and an arms race in outer space , and to use outer space for peaceful purposes . Basic obligations : not to test , deploy or use weapons , weapon systems or components . Consideration could also be given to an article on \u201c permissible activities \u201d thus helping to distinguish between activities that are prohibited and those that are not , and thereby safeguarding States Parties ' lawful right to utilize outer space for peaceful purposes . An article on definitions , providing clear definitions of the concepts mentioned , e. g. \u201c outer space \u201d , \u201c space weapons \u201d , \u201c weapon systems \u201d and \u201c components of weapon systems \u201d . Provision for appropriate national implementation measures and the designation or establishment of organizations to ensure that States
  Parties implement the instruments consistently and effectively . An article on international cooperation in the peaceful use of outer space promoting international exchanges , technical assistance and cooperation for peaceful purposes so that all countries can share in the economic and technological benefits of scientific advances in outer space , and outer space truly serves all mankind . Verification : we must first consider fully how technically feasible it is , and on that basis determine whether to use inspections or alternative means to prevent treaty violations . Establishment of an appropriate mechanism for consultations , clarifications and resolution of possible disputes in order to appropriately address such suspicions and disputes as might arise among States Parties . Appropriate , rational and workable confidence-building measures to enhance mutual trust among States Parties and forestall unnecessary suspicion about particular activities . The procedural articles commo
 nly found in international legal instruments dealing with amendment , length of validity , signature , ratification , entry into force , depository and authentic texts . These may of course also have to resolve some sensitive and key issues . The Chinese delegation wishes to emphasize that these are only tentative ideas that need to be developed . Our aim in putting them forward is to give all participants food for thought , pool our collective wisdom and encourage a fuller , more detailed examination of the relevant issues at the Conference . We will participate in such discussions and negotiations with an open mind , listening to and accepting good ideas and proposals from all parties and striving unremittingly to prevent the weaponization of and an arms race in outer space and to ensure the continued peaceful use of outer space for the benefit of all mankind . 
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/184un_CD_1672.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/184un_CD_1672.txt b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/184un_CD_1672.txt
new file mode 100644
index 0000000..58a89ff
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/txt/un/C/184un_CD_1672.txt
@@ -0,0 +1,2 @@
+
+28 May 2002 Original : SPANISH THE STATEMENT BY MR . FIDEL CASTRO RUZ , FIRST SECRETARY OF THE REPUBLIC OF CUBA , IN RESPONSE TO STATEMENTS MADE I have the honour to forward herewith the text of the statement made on 10 May 2002 by Fidel Castro Ruz , First Secretary of the Central Committee of the Communist Party of Cuba and President of the Councils of State and of Ministers of the Republic of Cuba , in response to statements made on 6 May 2002 by John Bolton , Under-Secretary of State of the United States of America . I should be grateful if you could arrange to have the enclosed text circulated as an official document of the Conference on Disarmament . ( Signed ) : Iv�n Mora Godoy Ambassador Permanent Representative \u201c There will always be weapons more powerful than those obtained through technology : the weapons of morality , reason and ideas \u201d Response by Commander-in-Chief Fidel Castro Ruz , First-Secretary of the Central Committee of the Communist Party of Cuba and Presid
 ent of the Councils of State and of Ministers , to statements made by the United States Government concerning biological weapons on 10 May 2002 , \u201c Year of the Heroic Prisoners of the Imperialists \u201d A mere three days ago , Otto Reich , Assistant Secretary of State , a person only too well known to us , was unmasked in an embarrassing lie when he said that four Cuban planes had landed in the Venezuelan capital on 12 April and : \u201c What they were doing there , what they were carrying , we do n't know . \u201d This was , it transpires , the beginning of an anti-Cuban campaign or an act of vengeance for the extraordinary debacle of the fascist coup which he had orchestrated , or both . When challenged publicly by the Cuban Ministry of Foreign Affairs on Tuesday , 7 May , the State Department said that it had no firm information , and did not wish to discuss the matter any further . The idea of destroying Cuba , an obsession which has lasted more than 43 years , has led and continues t
 o lead United States policy down a tortuous path of lies , fiascos and blunders . The discrepancy between what the United States Government is telling the world today and what it is actually doing with Cuba represents the most egregious and demoralizing contradiction of its foreign policy . Never before has this super-Power found itself in such a compromising situation , which has left it no choice but to lie , lie and lie again . There is no shortage of unscrupulous characters in high public office prepared to do this , or of press spokesmen painfully contorting themselves to disavow the wrongdoings of their superiors and to explain the inexplicable in their statements . Even a man like Colin Powell , the son of Jamaican immigrants , who , notwithstanding his military training - or perhaps because of it - is not considered to be a warmonger since he knows war and has seen men die , a man whom many Americans have come to look on as a potential presidential candidate , has found hims
 elf embroiled by these characters in squalid machinations that do him little credit . He better than anyone else knows how inexperienced they are and can see what little political and intellectual worth they have . As for this new character who has entered the scene with his sinister manoeuvres against Cuba , whom is he hoping to deceive ? We refer to John Bolton , Under-Secretary of State , responsible for nothing less than arms control . What exactly is this official hoping to achieve with his inflammatory and aggressive speech against Cuba given at the Heritage Foundation , well known for its ultra-right stance ? These statements , purportedly an analysis of the dangers of terrorism threatening the United States , began by saying : \u201c In addition to Libya and Syria , there is a threat coming from another signatory to the Biological Weapons Convention , and one that lies just 90 miles from the United States mainland - namely , Cuba . \u201d Then , after the usual name-calling and ab
 surd allegations filled with loathing that are so typical of arrogant and uninformed people , Mr Bolton added , this time off his own bat : \u201c We know that Cuba is collaborating with other State sponsors of terror . \u201c Castro has repeatedly denounced the United States ' war on terrorism . He continues to view terror as a legitimate tactic to further revolutionary objectives . Last year , Castro visited Iran , Syria and Libya - all designees on the same list of terrorist-sponsoring States . At Tehran University , these were his words : ` Iran and Cuba , in cooperation with each other , can bring America to its knees . The United States regime is very weak , and we are witnessing this weakness from close up . ' \u201c But Cuba 's threat to our security often has been underplayed . An official United States Government report in 1999 concluded that Cuba did not represent a significant military threat to the United States or the region . It went only so far as to say that ` Cuba has a lim
 ited capacity to engage in some military and intelligence activities which could pose a danger to United States citizens under some circumstances . ' \u201d Mr. Bolton then immediately started casting about for some explanation for the suspicious fact that no United States Government official had ever before thought to level such a deplorable accusation against Cuba . Bolton blames this weakness on William Cohen , who was United States Defence Secretary four years earlier when the much criticized report was issued . Bolton omitted to mention the fact that , a mere five months and two weeks before , on 19 November 2001 , he himself had not made the slightest reference to Cuba in his address to the Fifth Review Conference of the Parties to the Biological Weapons Convention in Geneva , when he cited many countries that were of concern to him as potential producers of biological weapons . So why this sudden and unexpected change ? Mr. Bolton 's scurrilous remarks of 6 May conclude with the
  allegation : \u201c For four decades Cuba has maintained a well developed and sophisticated biomedical industry , supported until 1990 by the Soviet Union . This industry is one of the most advanced in Latin America and leads in the production of pharmaceuticals and vaccines that are sold worldwide . Analysts and Cuban defectors have long cast suspicion on the activities conducted in these biomedical facilities . \u201c Here is what we now know : The United States believes that Cuba has at least a limited offensive biological warfare research-and-development effort . Cuba has provided dual-use biotechnology to other rogue States . We are concerned that such technology could support biological weapons programmes in those States . We call on Cuba to cease all biological weapons-applicable cooperation with rogue States and to comply fully with all of its obligations under the Biological Weapons Convention . \u201d Just as he had hoped , Mr. Bolton 's string of barefaced lies was immediately pi
 cked up by the international press . \u201c Washington , 6 May ( ANSA ) . Under-Secretary of State John Bolton today accused Cuba of helping ` enemy Governments ' with biological weapons programmes . \u201c ` Cuba has at least an offensive biological weapons programme and could be transferring its results to other States hostile to the United States ' , said Bolton . \u201c Under-Secretary Bolton was speaking to the Heritage Foundation , one of the ultra-conservative groups in Washington . \u201d \u201c Washington , 6 May ( DPA ) . The United States today accused Cuba of developing offensive biological weapons and of providing information about these weapons to States hostile to the United States , and urged President Fidel Castro 's Government to cease this cooperation . \u201d \u201c Washington , 6 May ( Reuters ) . On Monday , the United States accused three countries - Cuba , Libya and Syria - of trying to develop weapons of mass destruction and warned that it would take steps to prevent them from p
 roviding these arms to terrorist groups . \u201c ` States that sponsor terrorism and seek to obtain weapons of mass destruction must cease these activities ' , he said . ` States that renounce terror and abandon weapons of mass destruction can become part of our effort . But those which do not can expect to become our targets . ' \u201d \u201c Washington , 6 May ( EFE ) . The United States today added Cuba , Libya and Syria to the list of countries which form the ` axis of evil ' intent on manufacturing weapons of mass destruction , and warned that it would take steps to prevent them from supplying such weapons to terrorist organizations . \u201d \u201c Washington , 6 May ( Notimex ) . The United States today included Cuba on the so-called ` axis of evil ' , because it believed that it has the capacity to develop biological weapons which would pose a threat to its security , greater than that posed by Iraq , Iran and North Korea . \u201d \u201c Washington , 6 May ( AFP ) . On Monday , the United States 
 warned Cuba against any proliferation of biological weapons , urging the island 's Government to cease any supplies of biotechnological equipment to countries that Washington considers potentially dangerous , such as Iraq and Libya . \u201d The list of such articles and reports on this issue could be continued indefinitely . So , the deed has already been done ! The whole world , and in particular the American public , which is constantly bombarded with shameless lies , has been fed the necessary information and is now ready to believe that Cuba is a biological power , that it has a programme for the production of such weapons and that it poses a danger to the United States . And , as this has been stated by Mr. John Bolton , the very distinguished Under-Secretary of State for Arms Control and International Security , we have to believe it . Tired old tricks and stupid arguments As the old saying goes , it is easier to catch a liar than a lame man . Some people in the United States its
 elf find all this simply too hard to believe and are beginning to read between the lines . \u201c Washington , 7 May ( Notimex ) . The United States today refused to produce the evidence it claimed to have to back up accusations it had made against Cuba to the effect that the island had a limited capacity to develop chemical and biological weapons . \u201c The spokesmen of both the White House and the State Department maintained that the accusation against Cuba was not based on assumptions but on confidential information about the biological and chemical potential of the Cuban pharmaceutical industry . \u201c This accusation came as a surprise not only to the international community , but even to members of the United States Congress . According to the same dispatch , \u201c Political analysts pointed out that the accusation against Cuba by the Bush administration could form part of a White House strategy to justify moves to tighten its isolationist policy on Cuba . \u201c The statement that Cuba 
 could pose a terrorist threat to United States security was made at a time when several proposals to relax Washington 's Cuba policy were being studied on Capitol Hill . \u201c Political analysts believe , however , that , given this possibility , and bearing in mind that his brother Jeb will be seeking re-election as Governor of Florida this year , President Bush is hoping to ingratiate himself with the Cuban exile community . \u201c Given that more than four decades of economic blockade against Cuba have not succeeded in removing Fidel Castro from power in Havana , the only people who want this isolationist policy to remain in effect and to be tightened are the great majority of the Cuban exile community in the United States . \u201c The possibility of including Cuba as a member of the so-called ` axis of evil ' , with Iraq , Iran and North Korea at its head , would make it easier for Bush to obtain Congress backing for moves to increase - rather than decrease - the economic stranglehold w
 hich it has on the island . \u201d \u201c Washington , 7 May ( AP ) . ` I think that it will hold up the adoption of any new steps towards an opening up of trade , ' said Bob Graham , chairman of the Senate Select Committee on Intelligence . ` Unilateral steps will be most affected . ' \u201c Graham did not hide his amazement at Bolton 's accusation , however . He said that , in March , the Committee he chairs had held a secret hearing on security-related issues and that the administration had made no mention of biological weapons . \u201d On 7 May , when a journalist asked White House spokesman Ari Fleischer : \u201c Is there any proof of this or is this an assumption by the United States ? \u201d Fleischer replied : \u201c No , it 's not an assumption . I assure you that Secretary Bolton would not have said it if he did not have good cause , reasons and facts to base it on . It was based on sound analysis and on information available to the United States Government , which it has studied . \u201d This is
  a typical reply by someone who has no grounds or substance for what he says . The only things being carefully studied are falsehoods and deceitful ploys . One pities this spokesman for the job he has to do . Besides , why should we believe Mr. Bolton anyway ? Anyone who remembers the 15 incredible pretexts , which have come to light today in declassified official documents , that were devised at the end of 1961 by the powers that be in the United States with the aim of launching a direct military attack against Cuba in 1962 , would find nothing surprising in this sinister fabrication . But we demand proof . Let them produce even the smallest shred of evidence ! They do not have any and they cannot have any since there is not and cannot be any such evidence . As there is not so much as an atom of truth in what they say , they should not now try and hide behind the alleged sensitivity of their sources . This is a worn out old trick and an extremely stupid argument , which only serves
  to show in what very low esteem they hold the American public , whose intelligence deserves greater respect . I will also say this : If Cuban scientists from any of our biotechnology institutes had been cooperating with any country in the development of biological weapons , or had tried to create such weapons on their own initiative they would immediately be hauled before a court of justice on a charge of treason . The Acts of Terrorism Act passed by the National Assembly of Cuba stipulates , in its article 10 : \u201c Anyone who manufactures , facilitates , sells , transports , transfers , introduces into the country or keeps in his or her possession , under any form or in any place [ \u2026 ] chemical or biological agents , or any other substance from which any product that fits this description can be derived through research , design or a combination thereof \u201d shall be liable to penalties of between 10 and 30 years privation of liberty , life imprisonment or death . This is , witho
 ut question , a total fabrication , a treacherous blow against the sale of food to Cuba authorized by a law passed in 2000 , which has subsequently undergone countless modifications and amendments , introduced by those bitterly opposed to any lifting of the blockade , rendering it virtually unworkable , even when - despite everything - it was finally passed . Efforts are also being made to mislead and discourage the growing number of Americans who are increasingly repelled by the cruel and inhuman measures taken against the Cuban people , which fly in the face of the idealism and ethics of a nation which , for years and years , has been fed nothing but lies about Cuba . The only truth in Bolton 's falsehoods was the geographical fact that Cuba lies 90 miles from the continental territory of the United States . Slanderous invention , born of falsehood and deception It is false and a total distortion to assert that our country has repeatedly denounced the United States ' war against t
 errorism . I have said , and I stand by it , that the solution to this scourge will not be attained through war , which will serve only to breed hatred and fanaticism , but through sincere and determined cooperation among all countries in the world and by building a truly universal culture and mindset against terrorism . We were the first to suggest this form of cooperation , the very day of the tragedy in New York . It was a slanderous invention , born of falsehood and deception , to assert that Castro considers terrorism a legitimate tactic for furthering revolutionary objectives , since , as everyone knows , our revolutionary movement never practised any methods which were not consistent with our doctrine , our principles and our perception of armed struggle . You should get your facts straight , Mr. Bolton , and you should not be misled by the fictions narrated by your good friends in the Foundation . The civilian population and innocent people have never been the victims of our
  actions . Our tactic was always to fight against fully equipped enemy units . These days you seem to want to label as terrorism any armed resistance , regardless of the legitimate causes that may justify it . Applying this logic , you would be quite capable of describing as terrorism the struggle of the American colonists who rebelled and fought against English domination . George Washington and those who , after long years of war and enormous sacrifices , finally secured the independence of the United States were not terrorists . Your people led you astray , Mr. Bolton , when they told you about my address at the University of Tehran . It was not one address , but two , in two universities , and a few remarks which I made at the end of my visit to the mausoleum in which the body of the late Imam Khomeini lies . I have gone over them all in detail . In my address to the students in Tehran , there does not seem to be a single paragraph resembling that mendaciously cited in your stat
 ement on 6 May to the Heritage Foundation . At no point did I say that \u201c Iran and Cuba , in cooperation with each other , can bring America to its knees \u201d . I did say , in one of the three statements I made during the visit , that imperialism was bereft of ideas , that ideas were more powerful than weapons and that one day imperialism would collapse . I also said that the Shah \u201c was overthrown by the Iranian people through their heroism , and without weapons , which demonstrated the power of ideas and showed that there could never be any person in the world so powerful that he could not be brought down by ideas . This is our hope \u201d . And , at the end , I added : \u201c there is one super-Power which has thousands of nuclear weapons , aircraft , battleships , aircraft carriers , smart missiles [ \u2026 ] . However many weapons it may possess and however great its wealth , it will never be able to defeat the human spirit \u201d . I did not make the slightest allusion to the use of weap
 ons in this struggle . Indeed , I said just the opposite . That is what I think , that is how I saw it then and that is what I said . I am not in the habit of hiding my thoughts or twisting my words . My three statements in the Iranian capital were broadcast on Cuban radio and television . Thus , Mr. Bolton did not uncover anything , nor do I renounce any of my ideas . I said some other things about political beliefs and principles . I have all the tapes and the transcripts of those statements and I can prove what I said . I should also point out that , when I visited Iran , I had the honour of discovering a country with a culture thousands of years old , with deep religious convictions and great spiritual strength ; a country anxious to eradicate poverty , to fight against drug trafficking and other such plagues , a country determined to bring education , health , employment and well-being to more than 65 million people . Not one of the many leaders with whom I talked said anything
  to me about biological weapons or weapons of any other kind . I was able to see what an immense difference there is between the culture and customs of that country and those of the West . Iran was not the only country I visited : I went also to Algeria , Malaysia , Qatar , Syria and , finally , Libya , where I was shown how Qadaffi 's family home had been destroyed and I heard about the loss of human life - including that of a little girl - inflicted by the brutal F-16 air raid . Those parts of the world which I visited possess a wealth of history thousands of years old , which must not be destroyed and neither must their populations be annihilated . Thousands of millions of people throughout the world have watched with profound indignation the televised images of the horrendous events in Palestine . When he referred to my statement in Iran , Mr. Bolton irresponsibly and dishonestly mixed up excerpts of what I had said with bits and pieces of statements made by other people and rep
 orted by foreign press agencies and information from other sources picked up from wire-service cables or material simply concocted by himself . He made such a dog 's breakfast of all this material that he himself can no longer make head or tail of it . My clear , precise and transparent statements can be found in the speeches I have mentioned and in the press communiqu� of 10 May 2001 issued by both delegations , which , in paragraph 6 of the agreed document , sets out the Cuban position against terrorism , which states , and I quote : \u201c While condemning the phenomenon of terrorism in all its forms , especially State terrorism , the two parties agreed to cooperate closely at bilateral and international levels to fight against and eliminate this terrible phenomenon , to cooperate in efforts to combat international organized crime and drug trafficking , being always mindful of the cardinal principles of international law , and in particular those of sovereignty and non-interference
  in the internal affairs of States . \u201d This statement was issued 127 days before 11 September . I must thank Mr. Bolton for the praise which he heaps on our pharmaceutical industry : \u201c one of the most advanced in Latin America [ , which ] leads in the production of pharmaceuticals and vaccines that are sold worldwide , \u201d as he said in his own words . If only it were more advanced . Even his country could benefit from some of its attainments , but they would not permit it . He lies shamelessly , however , when he tells the American public and the world at large that the United States believes that Cuba is engaged in at least a limited offensive biological warfare research-and-development effort . This statement is as false as it is serious . Our researchers and doctors are educated in such a way as to instil in them a strong sense of solidarity and ethics . Millions of people in the world can testify to this . They work for the well-being and health of humankind . Over the last
  40 years , 34,307 Cuban doctors and health workers have provided their services , free of charge , in a large number of poor countries , saving lives and safeguarding the health of millions of people . No one in the world could surpass them in their dedication and willingness to make sacrifices . At this very time , 2,671 of them are working in isolated and inhospitable areas of Latin America , the Caribbean and Africa . It would be very difficult to persuade these men and women to manufacture viruses and bacteria to kill children , women , old people or the citizens of any country . The pride and high moral standards of our people , which have enabled them to withstand 43 years of attacks and blockade , rest on the sound basis of a policy which is entirely consistent with all their ethics and principles . In all , some 39,800 young people from more than 120 Third-World countries have graduated in Cuba in 33 specialized academic and technical fields . Even under the prevailing cond
 itions during this special period , induced by the cruel economic blockade , there are 8,053 young people from Latin America , the Caribbean and Africa following medical degree courses in our country - receiving training which , in the United States , costs more than $200,000 , without paying a single cent . Despite the lying allegations of Bolton and many others of his ilk , Cuba enjoys great recognition for the health services which it has rendered to humankind . These constitute a true fight against another form of terror , which many choose to ignore : the diseases which every year kill more than 11 million children , whose lives could be saved were it not for the selfish attitude of the industrialized world . An incalculable number of people survive or recover their health thanks to the selfless efforts of doctors from a small and poor country , who treat them with vaccines and preventive methods or therapies developed by Cuba . We would like to know if the United States Govern
 ment is doing something similar or if it would be prepared to cooperate in such programmes for the benefit of what it euphemistically terms the \u201c emerging countries \u201d , since we have not excluded any economically developed country from participating in these programmes . We have offered hundreds of scholarships at the Latin American School of Medical Sciences to American youngsters who lack the funds to study medicine in their own country . Cuban children are vaccinated against 13 diseases and enjoy splendid health . The infant mortality rate per 1,000 live births is lower than that of the United States . Medical care is guaranteed to 100 per cent of the population and is provided absolutely free of charge . The sorry situation in the United States , with its more than 280 million inhabitants , where 16 per cent of the population , including more than 10 million children , have no medical insurance , is quite unknown in Cuba . In such an immensely rich and scientifically advance
 d country , where hundreds of thousands of citizens die every year for these reasons , who is held to blame ? Who kills these people and who denounces these deplorable facts ? How can they concoct such lies ? With what moral right dare they affirm from over there that we , in Cuba , are developing biological warfare ? Who is going to believe their monstrous slanders ? On the other hand , never over the 43 years of its history since the revolution has anyone in our country launched or taken part in any terrorist act mounted from our territory against the United States . Not a single drop of United States blood has been shed , nor has any company lost so much as a simple screw as a result of any terrorist act hatched in Cuba . This cannot be said about Cuba by those in the United States who accuse our country of terrorism , or of supporting or sponsoring terrorism . Thousands of our compatriots have died and tens of thousands of acts of sabotage have been recorded as part of terrorist
  activities and aggression mounted from the United States against Cuba . How can their spokesmen deny these glaring facts ? I am not referring here to the American public : I am referring to their Governments . Not the slightest attempt has been made to clarify the overriding question of the powers granted to United States Government officials to conduct extrajudicial executions and to kill people anywhere in the world . I myself have often been the target of their sinister plans . That was how they operated in the past . Have they or have they not now reverted to those repugnant methods ? Why does Mr. Bolton not tell us something about that instead ? Cuba 's irreproachable policy on weapons of mass destruction Where weapons of mass destruction are concerned , Cuba 's policies have been irreproachable . No one has ever produced a single shred of evidence that our country has embarked on a programme to develop nuclear , chemical or biological weapons . Even people who have no compreh
 ension of the ethics , the attachment to the truth and the transparency which informs the conduct of a Government like that of Cuba will be able to understand at least that for Cuba to have acted in any other way would have been an act of immense stupidity . Any such programme would ruin the economy of a small country : Cuba would never have been able to transport such weapons and it would have been a further mistake to have used them in combat against an enemy which had thousands more such weapons and would have been only too delighted for an excuse to use them . From the political standpoint , we are living in an age that will bring weapons far more powerful than any that technology can produce : namely , the weapons of morality , reason and ideas . Without these , no country is powerful ; with them , no country is weak . To adhere to this maxim , any country needs to be extremely motivated , clear-headed and perspicacious . Let it be known that the Cuban people revere above all o
 ther things on earth the ideals that inspire freedom , dignity , love of their country , their identity and their culture , and the strictest possible adherence to justice . These are not weapons of mass destruction : they are weapons of mass moral defence and we are willing to fight and die for them . I understand that , for a man like Bolton , intoxicated with the military , economic and technological might of the super-Power on whose behalf he speaks , it might be difficult to understand these things . It would , however , be a good thing for him to try . Cuba has absolutely nothing to hide . On the contrary , it is proud of its progress in the field of biomedical research . Let us briefly review its history : 1979 : The Genetic Engineering Group was set up in the National Centre for Scientific Research ; 1981 : The first biotechnological scientific production institution designed for the manufacture of the Leukocyte Interferon was set up ; The pharmaceutical and biotechnological
  industry programme came into being as a result of the country 's scientific , economic and social development ; The necessary standards were set in place for inspection and certification by international agencies and the regulatory bodies of those countries with which Cuba was to have trade relations ; The period between 1990 and 1997 saw the development of the bulk of the investment programme , comprising 40 separate facilities ; An exclusively humanitarian industry is being developed , dedicated to the development and manufacture of medicines designed to prevent illnesses and to save lives , as well as to food production ; Technology and equipment used everywhere in the world is being purchased from commercial companies with international standing ; Under agreements entered into by Cuba with the Biological Weapons Convention , Cuba 's biotechnology , health system and civil defence sectors submit annual reports to the United Nations on confidence-building measures ; Regulatory bo
 dies from many different countries have visited Cuban biotechnology production facilities as a prerequisite for the marketing of our products in their countries ; Over the next few years , more than 50 new products , including biopharmaceuticals , vaccines and diagnostic kits , will come onto the market ; Cuba already possesses a wealth of intellectual property , comprising more than 150 inventions and over 500 patents registered abroad . The results of Cuban scientific research are published in leading international journals ; 1990 : First exports of Cuban products , which have increased every year since ; 1992 : Cuba signed the Convention on Biological Diversity and ratified it in 1994 ; 1995 : Landmark forms of marketing are introduced : technology transfers , risk contracts for development projects with foreign companies , manufacturing and trading partnerships ; Today , products and technology of the Cuban biotechnology industry are to be found in more than 40 countries . Techn
 ology transfer agreements or negotiations have been concluded or are under way with 14 countries : India : 4 transfers , 4 products ; China : 2 transfers , 4 products ; Brazil : 2 transfers , 2 products ; Egypt : 4 transfers , 1 product under negotiation ; Malaysia : 6 transfers ; Iran : 4 transfers , 4 products ; Russia : 1 transfer , 1 product ; South Africa : 1 transfer , 1 product ; Tunisia : 1 transfer , 1 product ; Algeria : 1 transfer , 3 products ; United Kingdom - Belgium : 1 transfer , 1 product ; Venezuela : 1 transfer , 2 products under negotiation ; Mexico : 1 transfer , 1 product . New trade and production negotiations are under way with 10 countries : Malaysia , Netherlands , Spain , Brazil , Venezuela , Viet Nam , Mexico , Ukraine , Germany and the United States ( negotiations are under way with the United States on the use of the Cuban anti-meningitis vaccine and the first steps have been made towards possible clinical trials with the EGF vaccine for lung cancer ) .
  Cuba 's biotechnology centres have already registered : 24 products , including both biopharmaceuticals and vaccines ; 49 state-of-the-art generic medicines ; 5 products for treating AIDS ; 15 new types of medical equipment ; 24 diagnostic systems . In addition to these achievements , scientists and researchers are continuing work on 60 more projects , prominent among which are the following : 29 new vaccines , including 8 cancer vaccines , 4 of which are already undergoing clinical trials , not only in Cuba but also in Canada , Argentina and the United Kingdom ; 21 innovative products for treating cancer , which supplement the 28 cytostatic drugs that have already been identified and which a new plant will soon begin manufacturing . Cuba 's biotechnology centres have applied for patents for 150 inventions , which will bring the total of such patents to 505. Four of these have been awarded medals by the World Intellectual Property Organization ( WIPO ) . Countless prominent politic
 ians , scientists and industrialists have visited Cuba 's biotechnology institutions . One of its main centres alone played host in 2000 to 1,520 visitors , 484 of whom came from the United States . The doors of our research centres are open to any international institution . In an official , public letter , Cuba has written to the Government of the United States to propose three major draft agreements , which are of greater benefit to the United States than to Cuba itself , given the very different magnitude of the problems in each of the two countries . The first is a draft agreement on immigration issues ; the second , a draft agreement on cooperation in controlling the illicit traffic in narcotic drugs and psychotropic substances ; and the third , a draft bilateral cooperation programme to combat terrorism . We have not received any reply . Does this mean that we are being accused of manufacturing biological weapons ? Whom do they hope to intimidate in this way ? We are urged to
  cease any biological weapons-related cooperation with \u201c rogue \u201d States and to comply with all our obligations under the Biological Weapons Convention . Which is the international organization that decides whether or not a country is a \u201c rogue \u201d State ? Precisely which rule of the Biological Weapons Convention has Cuba violated ? Is it that , on top of the criminal blockade , they would now ban us from marketing our medicines and using the most beneficial , the most noble products of the talents and hard work of our scientists , they would prevent us from placing these at the service of the health of people anywhere in the world ? Or could it be that the United States Government is seeking another bilateral agreement , in addition to those proposed by Cuba , namely , on cooperation in efforts to combat the production of biological weapons ? Please propose it . We would be happy to include it in our list of projects under consideration . We are sorry , Mr. Bolton . After the 
 lies , slander , fabrications and insults promulgated in your statement on 6 May , we regret to have to tell you that you have no moral ground to harangue Cuba on this subject , still less to do so in a threatening tone and language . Nor can you presume to give Cuba any lessons in politics and ethics . In any event , you and your Government should draw inspiration from Cuba 's decent and honourable behaviour . I can assure you that , for this technology transfer , we shall be making no charge . 
\ No newline at end of file


[42/51] [partial] opennlp-sandbox git commit: merge from bgalitsky's own git repo

Posted by bg...@apache.org.
http://git-wip-us.apache.org/repos/asf/opennlp-sandbox/blob/1f97041b/opennlp-similarity/src/test/resources/style_recognizer/all-tsne2.txt
----------------------------------------------------------------------
diff --git a/opennlp-similarity/src/test/resources/style_recognizer/all-tsne2.txt b/opennlp-similarity/src/test/resources/style_recognizer/all-tsne2.txt
new file mode 100644
index 0000000..e4fb507
--- /dev/null
+++ b/opennlp-similarity/src/test/resources/style_recognizer/all-tsne2.txt
@@ -0,0 +1,1051 @@
+<text id="syndicate/bmw-s-and-chinese-justice">
+BMW 's and Chinese Justice * * * * * In most places , the name " BMW " implies luxury and exclusivity . But in China , the brand has unwittingly found itself enmeshed in tales that illustrate the powerlessness of ordinary Chinese against the powerful and corrupt . The first scandal occurred last year , after a tractor filled with green onions scraped the side of a BMW in Harbin . The drivers of both the tractor and the metallic-silver luxury car were 45-year-old women , but any similarity between them ended there : the former was a peasant , the latter the wife of a wealthy businessman . After a confrontation between the two , the wealthy wife drove her BMW into the growing crowd of spectators on the roadside , killing the peasant woman and injuring 12 others . The case went to the local court , where the judge ruled it an " accidental traffic disturbance " and gave the driver a two-year suspended sentence . The judge 's ruling spurred rumors that he went easy on her because her hus
 band was related to senior provincial officials . It quickly became a story of " rich versus poor , " widely cited as an example of high-level corruption . Over the next few months , newspapers and television stations ran with the story as it snowballed into a national obsession . Alarmed by the public reaction , the Harbin magistrate ordered the case reopened , only to have the suspended sentence confirmed when the case was closed this March . The big BMW story of late 2003 resurfaced quietly , before being buried again forever . That same month , a lottery scandal threw the BMW brand into the limelight once again . Lottery officials in Shaanxi province rejected a winning ticket , calling it a fake and denying its bearer , a 17-year-old security guard named Liu Liang , the grand prize of a $58,000 BMW and 120,000 yuan ( $14,510 ) in cash . Liu became so angry about being accused of fraud and denied the car that he climbed atop a high advertising billboard and threatened to jump as 
 a show of innocence . But the story did n't end when police officers managed to talk him down . News broadcasts covered his continued insistence that he did not forge his ticket , along with the lottery center 's claims that their rejection of the ticket was legitimate . The police finally stepped in and , after a careful investigation , announced that they had found the true criminal : Yang Yongming , a private businessman whom the local lottery administration had contracted to organize ticket sales . Yang had conspired with the government officials directing the lottery , who were arrested for malfeasance , to fraudulently obtain the top prizes . In June Liu Liang finally got what he deserved - a BMW-325i sedan and a sincere apology from the lottery center . If the first scandal was a tragedy , the second was more like a farce . But both offer keys to understanding contemporary Chinese psychology . The outcry after the first BMW case was not really about the light sentence given t
 o a rich woman , but about the lack of confidence ordinary people have in China 's judicial system . In China , power , money , and connections trump the law . Even as they are becoming ever more litigious , many Chinese believe that they have no hope of securing justice against the powerful . The apathetic response of the dead peasant woman 's husband to the $10,000 in compensation he received was telling . " I do n't care about the verdict and whether it is justice or not , " he said . The most harmful consequence is the public 's loss of trust in the system . Social trust is not something you can buy with money . If an entire society believes that you cannot depend on legal rights for protection - that one must instead rely on a web of relationships with those who have power and influence - questions about whether such a society is livable or desirable will remain . Similarly , in the second BMW case , people did n't blame a corrupt businessman ; they ascribed culpability to the 
 lottery center , a government body . So instead of mistrusting one person , they grew suspicious of an institution - even of government itself . When a stubborn teenager went up against the mighty lottery authority with its army of auditors and inspectors and initial alibis , this individual , not the system , was the clear winner of the public 's admiration . Liu Liang may have been just a working-class kid , but there was wisdom in his words that there is still a " silent majority " who can affect the workings of China 's fragile society . He refused to settle privately , because he believed that if he let corrupt government officials off the hook , " they 'll keep scamming the public . " Thanks to his perseverance and the media 's investigation , the fraud was laid bare . Power corrupts everywhere , but individuals in China such as Liu have come to form a countervailing force . Even so , such marginal forces do not yet constitute a system for redressing grievances . While pop mus
 ic fans in China can listen to whatever they like , including Madonna singing " I 'm gonna shake up the system , " ordinary Chinese need courage to speak such messages aloud . As one saying goes , " There is not want of conscience in Chinese , but there is want of courage . " </text>
+<text id="syndicate/china-and-a-new-balance-of-power">
+China and a New Balance of Power SHANGHAI \u2013 The last rival superpower to the United States , the Soviet Union , collapsed in 1991. But , apart from its military strength , the USSR was never really powerful enough to counterbalance US influence . In the late 1980\u2019s , Japan seemed capable of challenging America\u2019s industrial leadership , but by the 1990\u2019s it lost its competitive edge . China might itself wish to be a major force in a multi-polar world , but has been plagued by its lack of overall strength . Given these realities , China sees the expanding European Union as a likely counterweight to unchecked US power . In terms of economic output , today\u2019s EU is on par with the US . But it has yet to build a strong defense system that can respond effectively either to regional contingencies or to global needs . What would be the strength of such a system if and when it is developed , and how will it compare with that of the US ? In assessing America\u2019s strength , China foll
 ows US debates on the merits of a uni-polar or a multi-polar world with keen interest . Some Americans favor a uni-polar system in which the US dominates . Such a Pax Americana would cost the country less to sustain , but the world would worry if America adopts a wrong policy , as has been the case in Iraq . The US certainly has a right to curb terrorists like those who staged the attack on September 11 , 2001. But the \u201c war on terror \u201d did not warrant the decision to attack a sovereign state and topple its government on the flawed presumption that it housed weapons of mass destruction and was linked to the 2001 attacks . The US went to war despite strong opposition from France , Germany , Russia , China , and other United Nations Security Council members . France and Germany have since sought to restore good relations with the US , but continue to disapprove of the war . The Middle East , always a region of concern for Europe , has become even less stable as a result of the war
  . American leadership has been one of the war\u2019s casualties . In China , the concept of an independent European defense \u2013 embodied in the European Security and Defense Policy ( ESDP ) , as well as the EU\u2019s Rapid Reaction Force ( RRF ) and various police missions \u2013 is thus generally seen as necessary and effective in a multi-polar world system . China observes that on most security matters , the ESDP will ( for now ) mimic American security interests , because the EU and US share fundamental values of human rights and democracy . Moreover , many EU members have simultaneously pledged their defense capabilities to NATO , which has been led by the US since its creation . But the ESDP , once fully developed , will not necessarily follow America\u2019s lead unconditionally , especially if American policy continues to deviate from the norm of international law , as in Iraq . As a result of acquiring a pan-European defense capacity , the Europeans are likely to play a more independent
  role than at present in managing intra-European security relations and carrying out global missions . China welcomes this expanded security role for the EU . Although it remains wary of international intervention by the EU , the substance and pattern of ESDP operations are likely to win China\u2019s respect for several reasons . First , China\u2019s leaders note that the ESDP gives priority to the legitimacy of its missions . So far , all ESDP missions have respected international law and governmental arrangements among disputing parties . Most of its military or police missions have been based on UN Security Council resolutions , and ESDP missions outside Europe have been at the invitation of local authorities . Of course , ESDP security missions , even those in accordance with Security Council resolutions , may not operate within the UN system \u2013 the EU prefers its own independent leadership . The ESDP does not necessarily require Security Council authorization as a condition for its 
 missions , and it retains an independent role in executing them . However , when compared with the US , the ESDP ( in Chinese eyes ) cares far more about international legitimacy , backed by the authority of the UN . Second , the ESDP is concerned with good governance and institution building . The EU is keen to restore or establish human rights , stability , and prosperity . Outside Europe , the ESDP acts to enhance governance rather than promote regime change , and the EU has often assisted governments \u2019 efforts to improve security conditions . Third , the ESDP is open to international cooperation . The EU works with either non-EU states , such as NATO members like Canada , Norway , and Turkey , or applicants for EU membership . The ESDP cooperates with the UN and other regional organizations , such as the African Union and ASEAN . When the EU reaches out , it tends to play a leading role in these collaborations . There are good reasons to expect that China will continue to acce
 pt an independent European security mechanism . There is little concern , if any , about the ESDP intervening in internal Chinese affairs , such as Taiwan . Indeed , China wants a strong and independent Europe , and from that vantage point it is not too early for China to envisage a truly multi-polar global system . </text>
+<text id="syndicate/china-and-russia-in-the-new-world-disorder">
+China and Russia in the New World Disorder Can Kosovo achieve independence without the tacit consent of Russia , and can there be a humanitarian and political solution to the tragedy in Darfur without the active goodwill of China ? The two crises have nothing in common , but their resolution will depend in large part on whether these two permanent members of the United Nations Security Council use their veto power . Comparing the respective abilities of Russia and China to block key international initiatives makes no sense in itself , but it does constitute a useful tool for understanding the transformation of the international system that is now taking place as a direct consequence of the relative decline of America\u2019s global power . From that standpoint , the deepening of chaos in the Middle East poses both opportunities and risks for Russia and China , which may force them to define the roles they want to play and the images they want to project in the world . The key question i
 s this : Is Russia taking giant steps in the \u201c wrong direction \u201d while China is taking \u201c minuscule \u201d steps in the \u201c right direction \u201d ? Superficially , Russia and China may give the impression that they are pursuing the same path when they both proclaim with pride that they are \u201c back \u201d on the world stage . But this boast means different things for each country . For China , a deeply self-confident country , to be \u201c back \u201d simply means regaining the country\u2019s historical centrality in the world after an absence of more than two centuries . After all , at the end of the eighteenth century , China became the world\u2019s first producer of manufactured goods , and it perceives itself as a center of civilization unequalled by any other in Asia , if not the world . China\u2019s renewed self-confidence is based on its remarkable economic prowess , which is derived not from natural resources , but from productivity and creativity . Whatever the huge political , social , and 
 economic tensions may exist , there is a \u201c feel good \u201d factor in China , a sense of progress , with the 2008 Olympics in Beijing figuring as the symbolic moment that will proclaim to the world the scale of the country\u2019s achievements . Above all , with the exception of the Taiwan issue , China is a satisfied status quo power when it comes to the evolution of the international system \u2013 a patient actor that finds it perfectly legitimate to behave and to be seen as the world\u2019s number two power . By contrast , the Russians remain insecure about their status in the world . Russia\u2019s explosive \u201c revisionist \u201d behavior on the eve of the recent G8 summit is an indication of the Kremlin\u2019s \u201c unsatisfied \u201d nature . Because they know they are less potent , particularly in demographic and economic terms , Russians feel they have to do \u201c more . \u201d For them , to say \u201c Russia is back \u201d means that the humiliating Yeltsin years are over , and that they now must be treated a
 s equals , particularly by the United States . Russians are nostalgic not for the Cold War as such , but for the international status they lost when it ended . Now that America is no longer a \u201c hyperpower \u201d with no strategic challengers , Russia has reasserted its status as a \u201c superpower , \u201d a claim that is not necessarily supported by reality . Unlike the Chinese , the Russians do not create economic wealth , but merely exploit their energy and mineral resources . Moreover , unlike the Chinese , they have not always been confident of their position in the world . Torn between Europe and Asia in cultural and political terms , victimized by a dark , narcissistic instinct that pervades their reading of their past and their visions of the future , it should surprise no one that Russia is now behaving like a \u201c revisionist \u201d power . Unsatisfied with their inner identity , it is only natural that Russians should demand changes that make them feel more secure and proud . Less 
 than 20 years ago , the Czech Republic and Poland were part of their sphere of influence , so Russians understandably cannot accept the US unilaterally implanting its security system there . Of course , in their respective judgments on Russia and China , the West \u2013 and Europeans in particular \u2013 may be demonstrating selective emotions . \u201c We \u201d tend to be less demanding of China than of Russia , because we tend to see Russia as \u201c European \u201d ( at least culturally ) . As a result , the culture of physical violence and verbal provocation that is gaining ground in Putin\u2019s Russia is deeply disturbing , whereas we tend to judge Chinese misdeeds with a greater sense of distance , if not indifference . But the differences between Russia and China today may prove to be less significant tomorrow if the deterioration in the Middle East imposes a sense of collective responsibility on all five permanent members of the UN Security Council . It is one thing for Russia and China to expl
 oit America\u2019s growing difficulties from Gaza to Kabul ; but it is quite another if the situation deteriorates to the point of general destabilization in the region . Indeed , the Middle East\u2019s deepening problems may impose a sense of restraint in Russia and China by forcing them to calculate not in terms of their global \u201c nuisance value \u201d vis-�-vis the West , but in terms of their ability to make a positive and stabilizing contribution to the world order . </text>
+<text id="syndicate/china-japan-protests-islands-by-liah-greenfeld">
+The Roots of Chinese/Japanese Rivalry BOSTON \u2013 The anti-Japan protests that continue to roil China are just another indication of the rise of a potent Chinese nationalism . After a century slowly fomenting among Chinese intellectuals , national sentiment has captured and redefined the consciousness of the Chinese people during the last two decades of China\u2019s economic boom . This mass national consciousness launched the Chinese colossus into global competition to achieve an international status commensurate with the country\u2019s vast capacities and the Chinese people\u2019s conception of their country\u2019s rightful place in the world . Rapidly , visibly , and inevitably , China has risen . Indeed , our era will likely be remembered as the time when a new global order , with China at the helm , was born . Competitive national consciousness \u2013 the consciousness that one\u2019s individual dignity is inseparably tied to the prestige of one\u2019s \u201c people \u201d \u2013 worked its way into the mind
 s of China\u2019s best and brightest between 1895 and 1905. In 1895 , China was defeated by Japan , a tiny aggressor whom the Chinese dismissively called wa ( the dwarf ) . China was already accustomed to rapacious Western powers squabbling over its riches , but had remained self-confident in the knowledge of these powers \u2019 irrelevance . However , the assault from Japan , a speck of dust in its own backyard , shattered this self-assurance and was experienced as a shocking and intolerable humiliation . Japan\u2019s triumph in 1905 over \u201c the Great White Power , \u201d Russia , repaired the damage to China\u2019s sense of dignity . From the Chinese point of view , Russia was a formidable European power , one feared by other Western powers . Its defeat , therefore , was seen as a successful Asian challenge to the West , in which China , its intellectuals felt , was represented by Japan . Japan thus became the focus of Chinese attention . Gentlemen-scholars , who would reform and staff the Chin
 ese army and civil service in the early decades of the twentieth century , went to study in Japan . The Revolution of 1911 was inspired by the example of Japan\u2019s Meiji Restoration ; and , because early-twentieth-century Japan was stridently nationalistic , the new China that emerged from its image was constructed on nationalist principles as well . Thus Japan became the significant \u201c other \u201d for China , the model that was imitated and the anti-model that was resented . Chinese nationalism borrowed from Japan its concept of the nation , including the very word by which it was expressed ( kuoming , from the Japanese kokumin ) . The Kuomintang ( the Chinese Nationalist Movement ) was explicitly inspired by Japan and fueled by repeated Japanese aggression . Paradoxically , but not unexpectedly , Mao Zedong\u2019s struggle against the Kuomintang was inspired by anti-Japanese nationalism as well . As was the case virtually everywhere else , communism in China was nationalism incarnate 
 . Mao\u2019s speech on the establishment of the People\u2019s Republic plainly expressed the nationalist agenda behind it . Calling the nation \u201c communist \u201d assured the new People\u2019s Republic of China of the Soviet Union\u2019s support , which was viewed by Mao as more reliable than that of the United States . But neither the Russian nor the Chinese Communists were ever unclear about the nationalist nature of their respective projects . The upper echelons of the bureaucracy and intelligentsia in Russia and China were self-consciously nationalist and , throughout Communist rule , shrewdly pursued the supreme nationalist goal : prestige \u2013 the power , naked and otherwise , to impose the nation\u2019s will on others . But national consciousness , particularly in China , was limited to a narrow elite , leaving the masses almost untouched . This changed dramatically with the Chinese government\u2019s restoration of a capitalist economy . Much like in Germany in the 1840\u2019s , when the appeal to p
 rivate enterprise converted the entire middle class to nationalism , the explicit definition of economic power as the central pillar of China\u2019s greatness awakened ordinary Chinese to nationalism\u2019s appeal . Hundreds of millions now see themselves as sharing in the nation\u2019s dignity , and are eager to contribute to it , as well as to defend it from insult . Competition for prestige , even when the contest is economic , is not a purely rational undertaking . So it should be no surprise that old injuries are likely to resurface . Some Chinese , especially those who are not economically successful , harp bitterly on Japan\u2019s past depredations . Despite China\u2019s embrace of capitalism and Japanese investment , Japan remains China\u2019s reviled other . Indeed , a professor in Beijing told me not long ago , \u201c Two in every ten Chinese dislike the US , but nine in every ten hate Japan . \u201d For the West , there is a silver lining in this nationalist rivalry : neither China nor Japan is 
 a rogue state , and , so long as their quarrels do not lead to the use of unconventional weapons , we may treat the friction between them as an internal Asian quarrel . Japan , moreover , is likely to let today\u2019s passions over disputed islands in the East China Sea cool down , despite the anti-Japanese outbursts in Chinese cities . But the West \u2013 and the US in particular \u2013 is new to dignity games � la Chinois . If it gets carried away and presumes to talk down to the 5,000-year-old culture of the Sages , the West could become the next object of China\u2019s nationalist resentment . </text>
+<text id="syndicate/china-s-cyber-warriors">
+China\u2019s Cyber-Warriors NEW DELHI \u2013 The world now accepts that protecting our atmosphere , hydrosphere , lithosphere , and biosphere \u2013 the \u201c global commons \u201d \u2013 is the responsibility of all countries . The same norm must apply to cyberspace , which is critical to our everyday life , economic well-being , and security . At a time when cyber attacks are increasing worldwide , US Secretary of State Hillary Clinton was right to declare that an attack on one nation\u2019s computer networks \u201c can be an attack on all . \u201d Indeed , the attacks are a reminder that , as a new part of the global commons , cyberspace already has come under threat . Cyberspace must be treated , along with outer space , international waters , and international airspace , as property held in common for the good of all . And , like ocean piracy and airplane hijacking , cyber-crime cannot be allowed to go unpunished if we are to safeguard our common assets and collective interests . Naming China among a ha
 ndful of countries that have stepped up Internet censorship , Clinton warned that \u201c a new information curtain is descending across much of the world . \u201d Her statement , with its allusion to the Cold War-era Iron Curtain , amounted to an implicit admission that the central assumption guiding US policy on China since the 1990\u2019s \u2013 that assisting China\u2019s economic rise would usher in greater political openness there \u2013 has gone awry . The strategy of using market forces and the Internet to open up a closed political system simply is not working . Indeed , the more economic power China has accumulated , the more adept it has become in extending censorship to cyberspace . If anything , China has proven that a country can blend control , coercion , and patronage to stymie the Internet\u2019s politically liberalizing elements . Through discreet but tough controls , Beijing pursues a policy of wai song , nei jin \u2013 relaxed on the outside , vigilant internally . Google is now crying f
 oul over \u201c a highly sophisticated and targeted attack on our corporate infrastructure originating from China . \u201d But , despite its corporate motto \u2013 \u201c Don\u2019t be evil \u201d \u2013 Google itself was instrumental in aiding online censorship in China , having custom-built a search engine that purges all references and Web sites that the Chinese government considers inappropriate . Now Google itself has become a victim of China\u2019s growing cyber prowess , in the same way that appeasement of Hitler boomeranged onto France and Britain . China deploys tens of thousands of \u201c cyber police \u201d to block Web sites , patrol cyber-cafes , monitor the use of cellular telephones , and track down Internet activists . But the threat to the new global commons comes not from what China does domestically . Rather , it comes from the way in which the know-how that China has gained in fashioning domestic cyber oversight is proving invaluable to it in its efforts to engage in cyber intrusion across it
 s frontiers . Canadian researchers have discovered a vast Chinese surveillance system called \u201c GhostNet , \u201d which can compromise computers in organizations abroad through booby-trapped e-mail messages that automatically scan and transfer documents to a digital storage facility in China . This is what happened when computers of the Tibetan government-in-exile in Dharamsala , India , were attacked last year . India\u2019s national security adviser recently complained that his office was targeted yet again by hackers . \u201c People seem to be fairly sure it was the Chinese , \u201d he said . Officials in Germany , Britain , and the US have acknowledged that hackers believed to be from China also have broken into their government and military networks . The state-sponsored transnational cyber threat is at two levels . The first is national , with the hackers largely interested in two objectives . One is to steal secrets and gain an asymmetrical advantage over another country . Cyber intrusi
 on in peacetime allows the prowler to read the content and understand the relative importance of different computer networks so that it knows what to disable in a conflict situation . The other objective is commercial : to pilfer intellectual property . The second level of cyber threat is against chosen individuals . The most common type of intrusion is an attempt to hack into e-mail accounts . The targets also can face Trojan-horse attacks by e-mail intended to breach their computers and allow the infiltrators to corrupt or transfer files remotely . To be sure , if a cyber attack is camouflaged , it is not easy to identify the country from which it originated . Through the use of so-called \u201c false-flag espionage \u201d and other methods , attacks can be routed through the computers of a third country . Just as some Chinese pharmaceutical firms exported to Africa spurious medicines with \u201c Made in India \u201d labels \u2013 a fact admitted by the Chinese government \u2013 some Chinese hacker
 s are known to have routed their cyber intrusion through computers in Russia , Iran , Cuba , and other countries . But , like their comrades in the pharmaceutical industry , such hackers tend to leave telltale signs . Then there are many cases in which the attacks have originated directly from China . It seems unlikely that these hackers , especially those engaged in cyber espionage , pilferage , and intimidation , are private individuals with no links to the Chinese government . It is more likely that they are tied to the People\u2019s Liberation Army . In war , this irregular contingent of hackers would become the vanguard behind which the PLA takes on the enemy . Systematic cyber attacks constitute a new frontier of asymmetrical warfare at a time when the world already confronts other unconventional threats , including transnational terrorism . With national security and prosperity now dependent on the safekeeping of cyberspace , cybercrime must be effectively countered as an intern
 ational priority . If not , cyberspace will become the new global-commons battlefield . </text>
+<text id="syndicate/china-s-gift-to-europe">
+China\u2019s Gift to Europe Europeans are wrong to be angry with China because its currency peg to the US dollar has boosted the euro against most currencies on foreign exchange markets . On the contrary , they should view the currency peg as a valuable gift . In New York and other American cities , European shoppers are on a spree . They\u2019re enjoying first hand the euro\u2019s enhanced purchasing power abroad . The currency peg has also helped make the euro into an important reserve currency , rivaling the dollar . Investments from all over the world are flowing into Europe as a result . It was predictable that China\u2019s currency peg would have this effect . It forced China to accumulate huge amounts of dollars to stabilize its currency and gave the euro an extra boost as the dollar depreciated . To protect itself from the declining greenback , China and other countries are converting a portion of their accumulated dollar reserves into euros . The International Monetary Fund estimates t
 hat the euro\u2019s share of foreign exchange reserves rose to 26.4 percent in third quarter of 2007 from 25.5 percent in the second quarter of 2007 , and 24.4 percent in the third quarter of 2006. The euro\u2019s becoming a major reserve currency bestows important economic advantages on the euro-zone economy . Europeans should thank the Chinese currency peg for the part it is playing in this . Of course , with an augmented supply of euros , China can be expected to make substantial investments in Europe , which also is a good thing , especially since , in the United States at least , China has demonstrated a preference for investments in financial institutions rocked by the sub-prime crisis . Europe has plenty of those . So far , China\u2019s biggest European investment has been in the Belgian financial company , Fortis N. V. , where Ping An Insurance recently acquired a 4.2 % stake , making it the largest shareholder in the company . China Development Bank also made an investment in Barcla
 ys to bolster its bid for ABN AMRO which , though it failed , helped the Dutch bank get the full value for its assets . This is only a start . Merrill Lynch predicts that , as currency reserves climb in countries like China , state-run investment funds will grow to $7.9 trillion , from $1.9 trillion now . China\u2019s currency peg has also come in handy for Europe in its fight to keep inflation in check , while its monetary policy remains constrained by the sub-prime crisis . Experts estimate that , in terms of guarding against inflation , the euro\u2019s appreciation has been equivalent to about 35 basis points rise in the interest rate . The strong euro has helped insulate Europe from skyrocketing oil and commodity prices , which are priced in dollars on the world markets . The big downside of the currency peg for Europe is supposed to be exports . That\u2019s what sent French President Nicolas Sarkozy and other high-level European officials scurrying off to China last November\u2014and why m
 any ordinary Europeans continue to think of the Chinese as not playing by the rules . This is protectionist thinking par excellence . What makes exports so special ? Why are they more important than cheap imports or the bargains that European shoppers get in New York ? Should Europe give up all the advantages of the currency peg just because it makes European exports more expensive ? Of course not ! Note that it was French President Sarkozy who went to China to protest the currency peg , not German Chancellor Angela Merkel . German exports are not being substantially damaged by the strong euro\u2014indeed , German costs are so competitive that exporters are confident they can cope with the euro at 1.50 dollars and even higher . This is not true for France-- but the answer is obvious : reduce costs and France too will be able to cope with the strong euro . In the final analysis , cutting costs is all the Europeans can do if they want to preserve the competitiveness of their exports . Th
 e Chinese are not going to change their exchange rate regime , which has strategic as well as economic motivations , just because the Europeans are unhappy with the currency peg . China has given the Europeans a gift . They should be grateful , rather than foolishly insist that China take it back . </text>
+<text id="syndicate/china-s-threat-to-russia">
+China\u2019s Threat to Russia Last week , Russia and China held joint military maneuvers in the presence of both Russian President Vladimir Putin and Chinese President Hu Jintao . But a new strategic alliance between the two countries is not likely , as it is China that poses the greatest strategic threat to Russia , although many in the Kremlin seem blind to this as they rattle sabers at America and the West . Indeed , China officially considers several regions in Russia\u2019s Far East to be only \u201c alienated \u201d from it . China\u2019s territorial claims on Russia are often noted in Chinese grade school geography textbooks , which include a number of Russian Far Eastern regions within China\u2019s borders . This is consistent with the Chinese strategic concept of \u201c vital space , \u201d which includes all spheres of a state\u2019s strategic activities , on land , at sea , under water , in the air , and in space . The dimensions of \u201c vital space \u201d are determined by a country\u2019s economic , sci
 entific , technical , social , and military capabilities \u2013 in essence , its \u201c total power . \u201d According to Chinese theorists , the \u201c vital space \u201d of great powers extends far beyond a state\u2019s borders , whereas the \u201c vital space \u201d of weak countries is limited to strategic boundaries that do not always correspond to the borders of their national territory . Today , China has territorial claims against 11 of its 24 neighbors , including India , Japan , Vietnam , and the Philippines , in addition to Russia . In China\u2019s relations with all of them , the potential use of military force was and remains an important factor . In September 2006 , China\u2019s People\u2019s Liberation Army ( PLA ) conducted an unprecedentedly large 10-day exercise , involving the Shenyang and Beijing military districts , the two most powerful of China\u2019s seven military districts . Shenyang abuts the Russian Armed Forces \u2019 Far Eastern District , and the Beijing district shares a border with Russia
 \u2019s Siberian Military District . During the exercises , Shenyang units advanced 1,000 kilometers into the Beijing district , where they engaged in joint war games . To military observers , the Beijing/Shenyang exercises seemed to be practice for a possible offensive operation against Russia , because exercises on such a scale are undertaken only at the final stage of a multi-year program to train troops to enact specific strategic and operational plans . The geography of the exercises , and the offensive nature of the tasks undertaken , leaves little doubt that Russia was cast in the role of \u201c potential adversary . \u201d Such a show of force is an ancient , traditional Chinese political technique . Paradoxically , these exercises were undertaken during a period when bilateral political and economic ties appeared on the surface to be at their highest point . Russia has an important place in Chinese geopolitical calculations , as a supplier of both modern weaponry and energy resource
 s needed to continue its modernization . Therefore , the Chinese are doing everything possible to strengthen their economic and political position in Russia , and to draw Russia into their sphere of influence . And China is succeeding , most importantly by consistently reinforcing Putin\u2019s anti-American and anti-Western agenda . While Beijing/Shenyang exercise should have indicated to Russian leaders that China\u2019s intentions toward Russia may not always be benign , Russia\u2019s political and military leadership seem not to sense any threat ; on the contrary , they continue to sell the Chinese advanced weapons . Russia\u2019s current diplomatic tilt , indeed , is clearly against its own long-term national security interests . China will never be interested in Russia\u2019s economic and political modernization , for it prefers Russia to remain a source of mineral and energy resources and a vast \u201c strategic rear \u201d in its looming challenge with the United States . Likewise , the Shanghai 
 Cooperation Organization ( SCO ) , which just concluded its annual meeting , is in China\u2019s eyes a tool of regional policy that helps strengthen China\u2019s influence and control over Central Asia\u2019s natural resources at the expense of Russia . No nation threatens China\u2019s land borders . China can solve its domestic problems , such as separatism , by itself . China is militarily self-sufficient and needs military cooperation under the SCO framework only in order to free its hands if any conflict should arise that affects its interests . In fact , conflict between Russia and China is possible precisely in Central Asia , given the clear differences in the two countries \u2019 economic and political interests in that region . Aside from control of the region\u2019s energy supplies , water has become a potential source of conflict , given China\u2019s serious shortages . Yet , while the Chinese clearly understand these contingencies and are preparing themselves to deal with them diplomatically 
 and militarily , the Kremlin remains myopically obsessed with the phantom threat of America . Thus , as the Kremlin dreams of re-establishing its domination over what Russians refer to as the \u201c near abroad \u201d ( Ukraine , Belarus , the Baltic countries , and the other post-Soviet states ) , China is increasingly looking at Russia as its own \u201c near abroad . \u201d Will the Kremlin finally wake up to this ? Thirty-six years ago , Richard Nixon and Mao Zedong turned world politics upside down , as both America and China realized that it was the Soviet Union , and not each other , that posed the greater threat . Vladimir Putin needs his own \u201c Nixon moment . \u201d Alienating the West is a foolish strategy when the greatest long term threat to Russia comes from the East . </text>
+<text id="syndicate/europe-s-vital-french-connection-by-kemal-dervi">
+Europe\u2019s Vital French Connection PARIS \u2013 In the debates raging over the future of the European Union and the eurozone , Germany always takes center stage . It has the largest economy , accounting for 28 % of eurozone GDP and 25 % of the eurozone\u2019s population . It is running a current-account surplus that is now larger than China\u2019s \u2013 indeed , the largest in the world in absolute value . And , while weighted majorities can overrule it on some issues , everyone acknowledges that little can be done in the eurozone unless Germany agrees . But the emphasis on Germany , though justified , should not lead to an underestimation of France\u2019s critical role . France not only accounts for roughly 22 % of eurozone GDP and 20 % of its population \u2013 behind only Germany \u2013 but also has the healthiest demography in the eurozone , whereas the German population is projected to decline over the next decade . At the same time , France\u2019s critical role reflects more than its size . Indeed , 
 in terms of influencing outcomes in Europe , France is as important as Germany , for three reasons . First , France is an indispensable link between southern and northern Europe at a time of growing economic and financial division between creditors and debtors ( a fissure that has begun to assume a cultural dimension ) . An active France can play a bridging role , leveraging its strong relationship with Germany ( a friendship that is a pillar of the EU ) and its proximity and cultural affinities to the Mediterranean . France is \u201c southern \u201d in its current-account deficit , but \u201c northern \u201d in its borrowing costs ( slightly above Germany\u2019s ) , owing partly to inflows of capital fleeing the south , as well as to modest but positive economic growth . Moreover , there is no perceived \u201c re-denomination \u201d risk affecting French assets , given markets \u2019 confidence that France will retain the euro . So , while France faces huge economic challenges , its northern and southern 
 features permit it to play a proactive role in the European project\u2019s survival . French President Fran�ois Hollande has already given a rather successful preview of this role , meeting German Chancellor Angela Merkel in Berlin on his first day in office , and , a month later , participating in a high-profile meeting with the Italian and Spanish prime ministers in Rome . Indeed , he took the lead in adding a \u201c growth pact \u201d to the \u201c stability pact \u201d that had been negotiated under Merkel\u2019s leadership . Second , France , under its new center-left government , must demonstrate that the \u201c European model \u201d of a market economy coupled with strong social solidarity can be reformed and strengthened , rather than abandoned \u2013 not just in Europe\u2019s more pragmatic north , but also in its more ideological south . French Socialists will not renounce their traditional commitments ; nor should they . But they now have the opportunity to contribute to the European model\u2019s renewa
 l . Under Hollande , France\u2019s Socialists favor achieving that renewal through a process of social dialogue that convinces rather than imposes , that focuses both on revenue measures and on boosting government efficiency , and that may adopt some of northern Europe\u2019s more successful \u201c flexicurity \u201d policies , which combine greater labor-market flexibility with strong social protection . The reforms should also introduce much greater individual choice , permitting solutions to retirement , education , health , and lifestyle issues that can be more easily tailored to citizens \u2019 specific circumstances and needs . The government of Hollande and Prime Minister Jean-Marc Ayrault has strong majorities at all executive and legislative levels , giving it a rare opportunity to act . If it can renew the European model at home , it will be able to project that success much more widely , particularly in southern Europe , in turn reinforcing confidence and belief in the EU , particularly
  among the young generation . The French center-left must lead in conceiving a vision for Europe in which solidarity and equity reinforce long-term economic strength . Finally , along with the United Kingdom among European countries , France retains more of a global role than Germany has yet acquired . While the United Nations Security Council is outdated in its structure , it still plays a crucial legitimizing role ; and France , not Germany , is a member . In many other international organizations as well , France punches above its weight . Similarly , while France exports much less than Germany outside the EU , many large French enterprises rival Germany\u2019s in global reach and technical know-how . And French is still a global language . In other words , France not only is a link between Europe\u2019s north and south , but also contributes substantially to linking Europe to the rest of the world . Europe needs a renewed vision and effective policies to realize that vision . France
 s Socialist-Green government can play a critical unifying role as Europeans confront their biggest challenge in decades . Its success will be highly consequential \u2013 not least for the political debate that will inform the outcome of Germany\u2019s elections in 2013. </text>
+<text id="syndicate/exchange-rate-disorder">
+Exchange Rate Disorder NEW YORK \u2013 Two troubling features of the ongoing economic recovery are the depressed nature of world trade and the early revival of international global payment imbalances . Estimates by the International Monetary Fund and the United Nations indicate that the volume of international trade in 2010 will still be 7 % to 8 % below its 2008 peak , while many or most countries , including industrial nations , are seeking to boost their current accounts . Indeed , if we believe the IMF\u2019s projections , the world economy\u2019s accumulated current-account surpluses would increase by almost $1 trillion between 2009 and 2012 ! This is , of course , impossible , as surpluses and deficits must be in balance for the world economy as a whole . It simply reflects the recessionary ( or deflationary ) force of weak global demand hanging over the world economy . Under these conditions , export-led growth by major economies is a threat to the world economy . This is true for Chi
 na , Germany ( as French Finance Minister Christine Lagarde has consistently reminded her neighbor ) , Japan , and the United States . Countries running surpluses must adopt expansionary policies and appreciate their currencies . More broadly , to the extent that major emerging-market countries will continue to lead the global recovery , they should reduce their current-account surpluses or even generate deficits to help , through increased imports , spread the benefits of their growth worldwide . But , while that implies that emerging-market currencies must strengthen , disorderly appreciations would do more harm than good . To use an American saying , it might mean throwing out the baby ( economic growth ) with the bathwater ( exchange rate appreciation ) . Consider China , which accounts for the largest share by far of world trade among emerging economies . Real appreciation of the renminbi is necessary for a balanced world economic recovery . But disorderly appreciation may seri
 ously affect China\u2019s economic growth by disrupting its export industries , which would generate major adverse effects on all of East Asia . China needs a major internal restructuring from exports and investments , its two engines of growth in past decades , to personal and government consumption ( education , health , and social protection in the latter case ) . But this restructuring will tend to reduce , not increase , import demand , as exports and investment are much more import-intensive than consumption . Moreover , a sharp appreciation of the renminbi could risk domestic deflation and a financial crisis . Chinese authorities certainly seem to have that interpretation of the roots of Japan\u2019s malaise in mind as they seek to avoid rapid revaluation . The only desirable scenario , therefore , is a Chinese economy that transmits its stimulus to the rest of the world mainly through rising imports generated by rapid economic growth ( i. e. , the income effect on import demand ) 
 , rather than by exchange-rate appreciation ( the substitution effect ) . This requires maintaining rapid growth while undertaking a major but necessarily gradual domestic restructuring , for which a smooth appreciation is much better suited . Now consider other major emerging markets . Here currency appreciation is already taking place , pushed by massive capital inflows since the second quarter of 2009 , and in some cases it can already be said to be excessive ( for example , in Brazil ) . These countries can , of course , resist upward pressure on their currencies by accumulating foreign-exchange reserves , like they did before the global financial crisis . The result is , of course , paradoxical : private funds that flow into these countries are recycled into US Treasury securities via investment of accumulated reserves . Why should emerging-market countries \u2019 central banks undertake this peculiar financial intermediation , which represents a major cost , as the yield of priva
 te funds is higher than that of reserves ? The implication here is that relying on free movement of capital to achieve exchange-rate appreciation and current-account deficits may generate a myriad of problems , including slower economic growth and the threat of asset bubbles and financial crises of their own . So , a more orderly way to induce current-account deficits without risking disruption of emerging economies \u2019 growth should be considered . One solution ( already advocated by some , including me , and adopted to some extent by a few countries ) is broader use of capital-account regulations . Surprisingly , however , this issue has been entirely absent from current global debates on financial reform . Fortunately , the IMF opened the door to discussion of this issue in a recent staff position paper . Equally important , a desirable global scenario is possibly one in which most developing countries run current-account deficits . But this requires major reforms in the global f
 inancial system to reduce the vulnerabilities that such deficits generated in the past , and that were reflected in major financial crises in the developing world . These past crises gave rise to a form of \u201c self-insurance \u201d among developing countries through reserve accumulation . This helped many of them weather the recent storm , but it also contributed to global payments imbalances . Recent IMF reforms are just a step in the direction of trying to create better financial instruments to help these countries . It is essential , in particular , to create reliable large-scale financing for developing countries during crises , through a mix of counter-cyclical issuance of SDRs and emergency financing without onerous conditions . </text>
+<text id="syndicate/france-and-germany-must-act-in-iraq">
+France and Germany Must Act in Iraq Next week , President Bush , President Chirac , and Chancellor Schroeder will meet on the cliffs of Normandy to mark the 60th anniversary of the D-Day invasions that led to the liberation of Europe . They should also seize the moment to liberate themselves from the bitterness that has divided the Atlantic Alliance over the Iraq War , argue Pierre Lellouche and Christoph Bertram . When the United States launched the war against Saddam Hussein 's Iraq , France and Germany rightly warned that the invasion could well end up worsening instability in the Middle East and increase the threat of radical Islamic terror . But now that America and its coalition have proven themselves to be incapable of bringing stability to Iraq on their own , the French and German governments can no longer rest and smugly say , " I told you so , " as the situation deteriorates even further . Both governments must now become seriously and fully engaged in what must now be a u
 nited Western effort . Of course , it would be best for everyone - France and Germany included - if the current US-led coalition were to succeed and Iraq could turn into a pillar of Middle East stability and modernization . Yet , however desirable this outcome , it is no longer likely ( if it ever was ) . Internal stability and economic recovery continue to be elusive , with the specter of civil war hovering ever closer . A divided Iraq at war with itself would be a disaster for the region , for America 's international credibility and authority , and for transatlantic relations . In an already fragile region , a major Arab country like Iraq would most likely prompt interventions by its nervous neighbors - of which there is no shortage - if it turns into a failed state . Instead of the rule of law , there would be the law of the mafia , of terrorist groups , and of tribal militias . Many of them are already in place . Instead of moving toward modernization and development , the regi
 on would be confirmed as a cockpit for conflict , a threat to itself and to the world beyond . Were the US forced to withdraw in failure , Islamic terrorist groups would claim a historic victory , promising more bloodshed for the West . Moreover , America might also once again retreat bitterly into isolationism , blaming her failure on the UN and on disloyal European allies . So the stakes are high in Iraq , for Europeans no less than for Americans . That is why it is time that even those who have been most skeptical about US policy in Iraq start to do something about it . Both Jacques Chirac and Gerhard Schroeder have repeatedly pronounced that what happens in Iraq is a strategic challenge to their countries . Yet their actions in response to this insight have been modest . The recent turmoil in Iraq has , on the contrary , strengthened those in both capitals who argue that there is now even less reason to get involved on the ground . At the very least , France and Germany are dema
 nding that the Bush administration openly admit to its failure and the need for help . Yet not only is such a demand na�ve , with the Bush team battling for re-election , it is not at all certain that most of America 's European allies would heed a US call for solidarity if one were issued . Confronting the threat that an unsettled Iraq poses to Europe and to the transatlantic relationship implies stopping these tactical games . The passing of authority in Iraq from the US-led coalition to an Iraqi government and the need for a much stronger UN role now offers the chance and imposes the need for everyone to stop posturing and get serious . For their part , EU governments should provide the UN - on whose involvement in Iraq they have long insisted - with the protective force it needs to prepare nation-wide elections in Iraq later in the year , and ask others to join . A UN staff protected by US marines simply cannot do that job . With the consent of both the US and Iraq 's new leade
 rship , EU governments should also offer to organize an international conference uniting all those with a stake in Iraqi stability , including Iraq 's neighbors , to work out a strategy and commit resources in order to save Iraq from slipping into protracted turmoil . If Europe is to act , France and Germany must lead . Precisely because they opposed the war , they are now the only countries that can restore Europe 's unity of action in promoting stability . France professes to bearing a special responsibility for international order , but so does Germany ; only a few weeks ago , Chancellor Schroeder based his claim to a permanent German seat in the UN Security Council on Germany 's readiness to shoulder such responsibilities . Of all the crisis regions in the world today , instability in Iraq constitutes the greatest challenge to international order . But France and Germany can also block European responsibility for international order and for solidarity with a US facing possible d
 efeat in Iraq . The decision is theirs . It is time for both to realize that the consequences of inaction will be no less severe for the region than for the Atlantic relationship . </text>
+<text id="syndicate/france-s-fourth-moment">
+France\u2019s Fourth Moment KYIV \u2013 Since World War II\u2019s end , France has consistently risen to the challenge of restructuring Europe in times of crisis . In doing so , France became the catalyst not only for building European unity , but also for creating the prosperity that marked Europe\u2019s post-war decades \u2013 a prosperity now under threat because of the global financial and economic crisis . If we are to see a stronger Europe emerge from today\u2019s challenges , visionary French leadership is needed again . The first moment when decisive French leadership began to unify Europe came when Robert Schuman and Konrad Adenauer created the European Coal and Steel Community . By rooting the then West Germany in the political , economic , and social fabric of the West , that step heralded the start of Germany\u2019s rebirth and economic miracle . The second time that France consolidated Europe\u2019s internal architecture came in 1983 , during the debates over the stationing of American cruise 
 and Pershing missiles in Germany to counter the deployment of SS20 missiles by the Soviet Union . France\u2019s resolve to stand with Germany helped prevent Germany from drifting into a dangerous neutrality that would have shaken the European community to its core . The third moment came after the Berlin Wall fell , and it was feared that an enlarged Germany might destabilize Europe . After initially trying to delay reunification , France embraced it , in exchange for Germany\u2019s reaffirming its commitment to European unity and Franco-German leadership of the European community . As a result , Germany bound itself to the idea of \u201c ever closer union \u201d even more definitively by pledging to join the common European currency , the euro . It is now urgent that the insights that have animated Franco-German relations be applied to all of Central and Eastern Europe . Only by securing the European identity of this entire region , and by anchoring the growing Russo-German relationship in a E
 uropean context , can the European Union continue on its path of stability and prosperity . Only by reinforcing open trading and financial relations across Europe can we hope to subdue today\u2019s crisis . For it is in no European country\u2019s interest , or in the interest of the EU as a whole , that Central and Eastern Europe feel that they have been cast adrift or that Germany and Russia fixate on each other in this time of crisis . As with Germany in the 1950\u2019s , the nature of Russia\u2019s links with its immediate neighbors is turning out to be the defining factor in shaping the country\u2019s international image . Many observers regard these relations as a signal not just to the region , but to the rest of the world , of the sort of power that Russia wishes to be . The issue is partly one of \u201c internationalism \u201d itself . In contrast to Europe \u2013 with its close-knit network of multilateral organizations through which states formulate and conduct much of their foreign policies \u2013 
 Russia is not accustomed to intensively cooperative international procedures . But keeping Russia at arm\u2019s length from Europe has only strengthened the sense of isolation that many Russians feel , tempting them to define the country\u2019s interests in ways that are irreconcilable with those of Europe . It has also heightened Russia\u2019s desire to construct a special , bilateral Russo-German relationship heedless of the context of the EU . European history during the past 60 years makes clear that the most promising approach to meeting the challenge of national reconciliation and stability is not to focus on specific contingencies , but to establish procedures that encourage orderly change . A singular vision has always animated this process : animosity between neighbors must not be allowed to fester , and the rule of law must reign not just within countries , but also between them . To believe that such a vision can work for Russia , Ukraine , and Europe is not wishful thinking , but
  rests on the successful experience of France and Germany in promoting amity . Indeed , the existence or absence of a framework of cooperation often determines whether a dispute mutates into a crisis . Such cooperative frameworks seek to reconcile national independence with regional interdependence , focusing political leaders \u2019 minds on prosperity for their people rather than on gaining unilateral advantages that ultimately impoverish and unsettle everyone . The first lesson of European unity is that times of crisis must draw the continent closer together , not divide it through protectionism , competitive devaluations , and expulsions of immigrants . Similarly , the euro must not be allowed to become an Iron Curtain that consigns non-members to a high-risk zone where investors dare not venture . For Ukraine , Europe can help by embracing the free-trade agreement that we are now negotiating . Coupled with our successful membership in the World Trade Organization , Ukraine would s
 tand to benefit when world and European trade begins to recover . Europe can also consider using various stabilization funds to help our economy through the crisis that we are all enduring . I do not ask for these things out of a narrow concern for my country\u2019s health . Just as the US Federal Reserve has engaged in credit and currency swaps with Brazil , Mexico , Singapore , and South Korea and other countries to ease their access to the dollars they need , the European Central Bank must offer such swap arrangements to Europe\u2019s non-euro countries in order for trade and production processes to continue . Yes , these are dark times , and all politicians want to protect their voters . But Europe\u2019s greatest postwar leaders understood that keeping the wider view of Europe in mind is the best way to achieve this goal . As at so many times before , now \u2013 with all of our economies in peril \u2013 is a moment for decisive French leadership . </text>
+<text id="syndicate/fran-ois-hollande-s-wrong-idea-of-france-by-brigitte-granville">
+Fran�ois Hollande\u2019s Wrong Idea of France LONDON \u2013 France\u2019s new president , Fran�ois Hollande , has achieved a remarkable series of political victories \u2013 at home and in Europe \u2013 since his election in May . Unfortunately , his streak of success will inevitably call forth an economic reckoning that will shock France\u2019s apparently unsuspecting citizens and doom the French elite\u2019s approach to the \u201c construction of Europe . \u201d Since winning the presidency , Hollande has won a parliamentary majority and pushed Germany toward accepting joint liability for eurozone countries \u2019 debts . But forebodings of crisis have become widespread in French business and economic circles . But the real danger \u2013 which even Hollande\u2019s sternest critics may be underestimating \u2013 is not so much his individual policy failings ( serious though they may be ) as his approach to the twin challenges posed by France\u2019s economic imbalances and the eurozone crisis . On each front separately , he
  might manage to muddle through ; together , they look likely to cement France\u2019s loss of competitiveness . Declining competitiveness is best captured in a single indicator : unit labor costs , which measure the average cost of labor per unit of output . In a monetary union , discrepancies in wage growth relative to productivity gains \u2013 that is , unit labor costs \u2013 will result in a chronic accumulation of trade surpluses or deficits . Since the euro\u2019s introduction , unit labor costs have risen dramatically faster in France than they have in Germany . According to Eurostat data published in April 2011 , the hourly labor cost in France was \u20ac34.2 , compared to \u20ac30.1 in Germany \u2013 and nearly 20 % higher than the eurozone average of \u20ac27.6 . France\u2019s current-account deficit has risen to more than 2 % of GDP , even as its economic growth has ground to a halt . The high cost of employing workers in France is due not so much to wages and benefits as it is to payroll taxes lev
 ied on employers . The entire French political class has long delighted in taxing labor to finance the country\u2019s generous welfare provisions , thus avoiding excessively high taxation of individuals \u2019 income and consumption \u2013 though that is about to come to an end as Hollande intends to slap a 75 % tax on incomes above \u20ac1 million . This is a version of the fallacy that taxing companies ( \u201c capital \u201d ) spares ordinary people ( \u201c workers \u201d ) . Of course , such taxes on firms are always passed on to households \u2013 usually through straightforward price hikes , and , in France , also through unemployment . High tax rates on labor \u2013 together with rigid regulation of hiring and firing \u2013 make employers extremely reluctant to recruit workers . As a result , France has had chronic long-term unemployment \u2013 forecast to reach 10.5 % by 2013 \u2013 for many years . Hollande\u2019s predecessor , Nicolas Sarkozy , tried to address this problem . He exempted voluntary overtime pay fro
 m employment tax and shifted some of the burden of labor taxation onto consumption ( via a hike in VAT ) . But Hollande quickly reversed both of these reforms . The repeal of the tax break on overtime reflects another economic fallacy to which French Socialist politicians are deeply attached : the \u201c lump of labor \u201d notion that underlay the most disastrous of their economic policies \u2013 the 35-hour workweek , introduced in 2000. The idea behind the policy is that demand for labor is a constant , and that this fixed number of aggregate working hours required by employers to meet final demand can be spread more evenly among workers to reduce unemployment . Such measures , designed to create jobs by freeing up work hours , are futile at best , and are often detrimental . French Socialists should recall their school physics lesson about communicating vessels : when a homogeneous liquid is poured into a set of connected containers , it settles at the same level in all of them , regard
 less of their shape and volume . Generating more \u201c liquid \u201d ( jobs ) requires not discouraging the entrepreneurs on whose activities sustainable job creation ultimately depends . The effect of fiscal and regulatory pressure on employment is to encourage French firms to invest and hire outside France . Hollande\u2019s apologists praise his gradualist and consensual approach to addressing the economy\u2019s structural distortions . They argue that his penchant for setting up consultative commissions is the best way to forge the consensus required for structural reform , whereas Sarkozy\u2019s combative style was counterproductive . Even banishing skepticism and assuming that Hollande could over time persuade his supporters to embrace competitiveness-boosting policies , the eurozone crisis is denying France the time that such gradualism requires . A simple , effective way to buy time would be to abandon the euro and restore competitiveness through a devalued national currency . But this exp
 edient is incompatible with mainstream French politicians \u2019 devotion to the \u201c European project , \u201d which amounts to a projection of French soft power ; indeed , building Europe lies at the heart of the French establishment\u2019s version of what Charles de Gaulle used to call \u201c a certain idea of France . \u201d For mainstream French politicians , renouncing the European project to buy the time required to restore competitiveness is as unthinkable as is the logical alternative : an all-out push for full European political union . This would reestablish monetary sovereignty and create a normal central bank ( like the Federal Reserve or the Bank of England ) at the European level . But it would also mean abandoning France\u2019s republic in favor of a federal European government \u2013 anathema to that \u201c certain idea of France . \u201d The combination of gradualism ( on the most generous interpretation ) in domestic economic reform and the paralyzing effect of the eurozone crisis will lead 
 to a massive shock . Remaining in a currency union with the much more competitive German economy will require wrenching and rapid reforms , for which Hollande\u2019s tepid approach will fail to prepare the complacent French . The result will be even more support than was seen in last April\u2019s presidential election for extremist political parties that reject both Europe and competitive market capitalism . </text>
+<text id="syndicate/hollywood-s-favorite-villains">
+Hollywood\u2019s Favorite Villains Once upon a time , Cold War enemies , white supremacists , and evil geniuses reigned supreme as Hollywood\u2019s favorite bad guys . No more . Today , it is multinational corporations that are increasingly being cast as the �ber-villains of our globalized world . For all their subliminal paid promotions and subtle product placements , corporations are getting drubbed in the main story lines of our popular culture . This treatment goes far beyond documentaries like Michael Moore\u2019s polemical Fahrenheit 9/11 or The Corporation , an earnest if somewhat paranoid portrayal of multinational companies \u2019 role in globalization . It extends to mainstream hits like The Constant Gardener , in which the idealistic protagonists do battle with a malicious global pharmaceutical company that is bent on exploiting Africa\u2019s misery to test experimental drugs . To be sure , sociopathic corporations have populated books and films for more than a century . But corporate 
 villains , typically multinational companies , have never been so ubiquitous as today . Is it unfair ? Most corporations , after all , are merely convenient mechanisms for ensuring that scarce global capital is used at maximum efficiency , to the benefit of all . Are famously liberal Hollywood film directors spending too much time going to anti-globalization rallies ? Perhaps . But I would submit that Hollywood\u2019s misgivings , however untutored , represent only the tip of a growing iceberg of resentment against the perceived injustices of globalization . The simple truth is that corporations represent capital , and capital \u2013 in the form of factories , equipment , machines , money , and even houses \u2013 has been the single biggest winner in the modern era of globalization . Corporate profits are bursting at the seams of investors \u2019 expectations in virtually every corner of the world . Even in moribund economies like Germany and Italy , where employment security is vanishing , cor
 porations are swimming in cash . This phenomenon comes as no surprise to economists . Add two billion Indian and Chinese workers to the global labor force , and the value of other means of production \u2013 particularly capital and commodities ( for example , gold and oil ) \u2013 is bound to go up . And so it has , with capitalists everywhere gaining an ever larger share of the economic pie . ( In theory , capitalists in labor-abundant China and India could end up as losers , but in practice they , too , have benefited thanks to their governments \u2019 success in simultaneously liberalizing and globalizing . ) Many policymakers seem to be under the impression that surging profits are a purely cyclical phenomenon , as economies continue to grow out from the depths of the 2001 recession . Wait a bit , they predict , and wages will fully catch up later in the cycle . Not likely . Capital\u2019s piece of the pie has been getting bigger for more than 20 years , and the trend looks set to continue 
 . Indeed , corporations \u2019 growing share of income has been a major driver behind the long , if uneven , bull market in stocks that began in the early 1990\u2019s . At the same time , inflation-adjusted wages for rich-country unskilled workers have barely budged over the last two decades . Some of these trends also have to do with the nature of modern technological change , which seems to favor capital and skilled workers disproportionately . But , regardless of their cause , rapidly growing inequalities are a powerful force for instability everywhere , from wealthy America to rapidly growing China to reform-challenged Europe . \u201c A rising tide lifts all boats , \u201d conservatives like to say . Fine , but what happens to people , like the poor of hurricane-struck New Orleans , who don\u2019t own boats ? Growing inequality would not be such a problem if governments could simply raise taxes on the rich and strengthen subsidies to the poor . Unfortunately , any country that taxes capital to
 o aggressively will only succeed in chasing it to regions where the tax burden is lighter . In a globalized world , national governments \u2019 ability to tax potentially mobile factors of production is sharply circumscribed . The same mechanism that pours profits into the pockets of global corporations also prevents governments from claiming a larger share of the spoils . Unfortunately , the long-term trend towards ever-lower income shares for unskilled workers is likely to continue over the coming decades , as modern technology permeates the globe , and as emerging markets like China , India , Brazil , and Eastern Europe continue to integrate into global production . This is not to say that unskilled workers are actually being made worse off by globalization in absolute terms ; the vast majority are actually breaking even or better . But unskilled workers \u2019 incomes are not keeping pace with overall economic growth , and the resulting social strains are a ticking bomb . If so , then
  Hollywood\u2019s cartoon-like caricatures of evil multinational corporations may some day seize mainstream consciousness , leading to political upheavals that shatter today\u2019s social contract . That won\u2019t be good for profits , or for the poor . Governments \u2013 and corporations \u2013 must find better ways to provide equal opportunity through improved education , broader financial markets , and other channels . Otherwise , globalization\u2019s storyline may not proceed according to the script . </text>
+<text id="syndicate/japanese-lessons-for-china-s-currency">
+Japanese Lessons for China\u2019s Currency SHANGHAI \u2013 The question of how much China\u2019s currency should appreciate to rebalance its trade has become a global hot-button issue . But the answers have been all over the map , with some finding that the yuan is not undervalued at all , while others argue that it should appreciate against the dollar by more than 30 % . Clearly , there must be major differences in the macroeconomic models used to produce such a wide range of estimates . But the one thing about which everyone seems to agree is the theoretically and empirically unjustified assumption that an equilibrium exchange rate actually exists . The theoretical problem is simple : a country\u2019s trade balance depends on a lot more than the value of its currency in the foreign exchange markets . Interest rates , employment , aggregate demand , and technological and institutional innovation all play a role . As the economist Joan Robinson pointed out in 1947 , just about any exchange rate
  will be the equilibrium value for some combination of these other variables . The equilibrium exchange rate , she famously argued , is a chimera . Not surprisingly , the empirical evidence that trade imbalances can be resolved through exchange rate changes alone is unconvincing . In the case of China , the most useful precedent is probably that of Japan in the period from the end of the Bretton Woods fixed exchange-rate regime in August , 1971 , to the collapse of its \u201c bubble economy \u201d in 1990. During that period , the yen\u2019s value more than doubled against the dollar , rising from its original fixed rate of 360 to 144 at the end of 1989. Yet , even as Japan\u2019s exports became much more expensive in dollar terms and its imports much cheaper in yen , its trade surplus rose from $6 billion in 1971 to $80 billion in 1989. For two decades , expectations that an appreciating yen would restore external balance were repeatedly disappointed . At the time of the December 1971 Smithson
 ian Agreement , 308 yen to the dollar was supposed to do the trick . Fourteen years later , during the Plaza Accord negotiations , the Japanese argued for an eventual level of 200-210 , while some US Treasury officials thought the final target should be as high as 165-170 . At the end of the 1980\u2019s , some analysts thought rates as high as 120 might finally produce the long-sought equilibrium . Yet , as Japan entered the \u201c lost decade \u201d of the 1990\u2019s , its exports continued to grow faster than its imports . Japan\u2019s trade surplus peaked only in 1994 , at $144 billion , just a few months before the yen\u2019s April 1995 all-time high of 79.75 . In retrospect , it is easy to see why none of these supposed equilibrium exchange rates delivered external balance . As the yen appreciated , Japan responded not by exporting less but by improving productivity and quality control through plant and equipment investment and innovations in factory management , making possible rapid growth in
  exports of high-value-added products . Exchange-rate equilibrium calculations from the 1970\u2019s and 1980\u2019s , which could only have been based on the export sector\u2019s contemporary structure , naturally would have little relevance subsequently . The same is true of calculations at the beginning of the 1990\u2019s , which would have forecast Japanese import growth based on extrapolations from the high GDP growth rates of the previous 40 years rather than on the decade of stagnation that ensued . In China , changes in the export sector\u2019s structure similar to those observed in Japan are now taking place . These changes are likely to make today\u2019s attempts to find an equilibrium yuan-dollar exchange rate seem just as chimerical in hindsight as previous calculations of the yen-dollar equilibrium rate . For the 30 years since the beginning of China\u2019s economic reforms , Chinese industry has achieved impressive efficiency gains by adopting new technologies and realizing economies of sca
 le , leading to a huge expansion in locally made products suitable for export . While an appreciating currency might eventually drive labor-intensive manufacturers out of business , if Japan\u2019s economic history is any guide , they are likely to be replaced by producers of things like ships , machine tools , semiconductors , and doubtless new products yet to be invented . The equilibrium yuan-dollar rate is a chimera not because China\u2019s trade could never be balanced , but because the exchange rate alone does not determine equilibrium . The structure of the entire economy matters , too . As this is constantly evolving in unpredictable ways , there is no reason to expect that the assumptions underlying any particular macroeconomic model will ever remain valid long enough for its steady-state solution to be achieved in practice . </text>
+<text id="syndicate/kennedy--khrushchev--the-cuban-missile-crisis-and-nuclear-deterrence-from-cuba-to-iran-by-joseph-s--nye">
+The Cuban Missile Crisis at 50 CAMBRIDGE \u2013 This month marks the 50th anniversary of the Cuban missile crisis \u2013 those 13 days in October 1962 that were probably the closest the world has come to a major nuclear war . President John F. Kennedy had publicly warned the Soviet Union [ 178]not to introduce offensive missiles into Cuba . But Soviet leader Nikita Khrushchev decided to cross Kennedy\u2019s red line surreptitiously and confront the Americans with a fait accompli . When an American surveillance plane discovered the missiles , the crisis erupted . Some of Kennedy\u2019s advisers urged an air strike and invasion to destroy the missiles . Kennedy mobilized troops , but also bought time by [ 179]announcing a naval blockade of Cuba . The crisis subsided when Soviet ships carrying additional missiles turned back , and Khrushchev agreed to remove the existing missiles from the island . As then US Secretary of State Dean Rusk put it : \u201c We were eyeball to eyeball , and I think the oth
 er fellow just blinked . \u201d At first glance , this was a rational and predictable outcome . The United States had a 17-to-1 advantage in nuclear weaponry . The Soviets were simply outgunned . And yet the US did not preemptively attack Soviet missile sites , which were relatively vulnerable , because the risk that even one or two of the Soviet missiles would be fired at an American city was enough to deter a first strike . In addition , both Kennedy and Khrushchev feared that rational strategies and careful calculation might spin out of control . Khrushchev offered a vivid metaphor in one of his letters to Kennedy : [ 180]\u201cWe and you ought not now to pull on the ends of the rope in which you have tied the knot of war . \u201d In 1987 , I was part of a group of scholars that met at Harvard University with Kennedy\u2019s surviving advisers to study the crisis . Robert McNamara , Kennedy\u2019s secretary of defense , said he became more cautious as the crisis unfolded . At the time , he thoug
 ht that the probability of nuclear war resulting from the crisis might have been one in 50 ( though he rated the risk much higher after he learned in the 1990\u2019s that the Soviets had already delivered nuclear weapons to Cuba ) . Douglas Dillon , Kennedy\u2019s treasury secretary , said he thought that the risk of nuclear war had been about zero . He did not see how the situation could possibly have escalated to nuclear war , and thus had been willing to push the Soviets harder and to take more risks than McNamara was . General Maxwell Taylor , the chairman of the Joint Chiefs of Staff , also believed that the risk of nuclear war was low , and he complained that the US let the Soviet Union off too easily . He felt that the Americans should have removed the Castro regime . But the risks of losing control of the situation weighed heavily on Kennedy , too , which is why he took a more prudent position than some of his advisers would have liked . The moral of the story is that a little nuc
 lear deterrence goes a long way . Nonetheless , there are still ambiguities about the missile crisis that make it difficult to attribute the outcome entirely to the nuclear component . The public consensus was that the US won . But how much the US won , and why it won , is hard to determine . There are at least two possible explanations of the outcome , in addition to Soviet acquiescence to America\u2019s superior nuclear firepower . One focuses on the importance of the two superpowers \u2019 relative stakes in the crisis : the US not only had a greater stake in neighboring Cuba than the Soviets did , but could also bring conventional forces to bear . The naval blockade and the possibility of a US invasion strengthened the credibility of American deterrence , placing the psychological burden on the Soviets . The other explanation questions the very premise that the Cuban missile crisis was an outright US victory . The Americans had three options : a \u201c shoot-out \u201d ( bomb the missile si
 tes ) ; a \u201c squeeze out \u201d ( blockade Cuba to convince the Soviets to withdraw the missiles ) ; and a \u201c buyout \u201d ( give the Soviets something they want ) . For a long time , the participants said little about the buyout aspects of the solution . But subsequent evidence suggests that a quiet US promise to remove its obsolete missiles from Turkey and Italy was probably more important than was thought at the time ( the US also gave a public assurance that it would not invade Cuba ) . We can conclude that nuclear deterrence mattered in the crisis , and that the nuclear dimension certainly figured in Kennedy\u2019s thinking . But it was not the ratio of nuclear weapons that mattered so much as the fear that even a few nuclear weapons would wreak intolerable devastation . How real were these risks ? On October 27 , 1962 , just after Soviet forces in Cuba shot down a US surveillance plane ( killing the pilot ) , a similar plane taking routine air samples near Alaska inadvertently viola
 ted Soviet air space in Siberia . Fortunately , it was not shot down . But , even more serious , unbeknownst to the Americans , Soviet forces in Cuba had been instructed to repel a US invasion , and had been authorized to use their tactical nuclear weapons to do so . It is hard to imagine that such a nuclear attack would have remained merely tactical . Kenneth Waltz , an American scholar , recently published an article entitled [ 181]\u201cWhy Iran Should Get the Bomb . \u201d In a rational , predictable world , such an outcome might produce stability . In the real world , the Cuban missile crisis suggests that it might not . As McNamara put it , \u201c We lucked out . \u201d </text>
+<text id="syndicate/merkel-in-china-by-sanjaya-baru">
+Merkel in China NEW DELHI \u2013 German Chancellor Angela Merkel\u2019s second visit to China in a year comes against the backdrop of dire forecasts of a difficult September for the eurozone . Mindful of such concerns and persistent pessimism in global financial markets , Merkel is now taking bold political initiatives at home and overseas . Indeed , her China trip should be seen as an effort to assert leadership across the eurozone . At home , Merkel recently sent out a clear message to her critics that Germany must pay a price for eurozone leadership . She cautioned her colleagues against loose talk about a \u201c Grexit \u201d \u2013 Greece\u2019s exit from the eurozone \u2013 and assured visiting Greek Prime Minister Antonis Samaras that Germany remained committed to his country\u2019s membership of the eurozone . While it required courage to take such a tough stance , doing so helped to bolster her position at home and throughout the eurozone . There is now no doubt that Merkel is willing to commit Ge
 rmany to the cause of preserving both the European Union and the eurozone , and that she will work to achieve that goal . If she succeeds , she will emerge as the first great European leader of the twenty-first century . This stance suggests that Merkel appreciates the essence of the argument that Fran�ois Heisbourg , the chairman of the council of the International Institute for Strategic Studies ( IISS ) , advanced in a [ 179]recent essay : a federal arrangement does not fall apart because of problems at the periphery but because of \u201c failure at the heart of the system . \u201d Merkel has come to terms with Germany\u2019s position \u2013 and that it must act to preserve the unity of the whole . The eurozone ( and probably the EU ) cannot be saved in Greece or Finland if it cannot be saved in Germany . Committing Germany to that objective is precisely what Merkel has defined as her political goal for the rest of her term in office . Within hours of declaring her leadership at home , Merk
 el announced her visit to Beijing . The timing was highly significant : Merkel goes to China , the emerging global power , after declaring her own commitment to strengthening the eurozone , if not the EU as a whole , as a credible pole of an emerging multipolar world . In recognition of Germany\u2019s growing significance , Chinese Prime Minister Wen Jiabao welcomed a delegation of German business leaders to Beijing this May with the words : \u201c Stronger Chinese-German cooperation is good for the two countries , good for China-EU relations , and good for world prosperity and stability . The strategic dimension of Chinese-German relations , therefore , can only be strengthened , not weakened . \u201d There are several dimensions to the bilateral relationship . Germany needs both China\u2019s markets and the funds that its government can deploy to purchase German and European bonds . It also has more than 7,500 enterprises operating in China , with gross investment totaling $18.5 billion . Mor
 eover , Germany has sold $15 billion worth of technology to China , and bilateral trade hit a high of $169 billion in 2011 , accounting for 30 % of total China-EU trade . The two countries have set a bilateral trade target of $280 billion for 2015. Trade , however , is not the only immediate concern . Far more important , especially for Germany , is to get China to invest in and hold its bonds . In mid-August , at the First IISS Oberoi Lecture in Mumbai , Klaus Regling , the CEO of the European Financial Stability Facility , underscored the importance of Chinese demand for EFSF bonds and China\u2019s role in stabilizing the eurozone . Regling also revealed that there is now increased and frequent coordination between monetary authorities in the United States , Germany , and China , drawing attention to the fact that the \u201c old Triad \u201d of the dollar , euro , and yen may now have been replaced by a \u201c new Triad \u201d of the dollar , euro , and Chinese renminbi . While Regling spoke of 
 an emerging \u201c multipolar monetary system , \u201d his remarks clearly indicated the functioning of a \u201c tripolar \u201d system . Lying at two ends of that monetary triangle , the eurozone and China have acquired a geopolitical stake in helping each other . [ 180]Hans Kundnani and Jonas Parello-Plesner of the European Council on Foreign Relations view the China-Germany relationship as one that will shape the overall China-EU relationship . Like many strategic analysts , they worry whether Germany , in pursuit of purely short-term economic interests , might forsake long-term strategic interests and concerns about human rights , environmental problems , press freedoms , and other political and geopolitical issues . It is significant , therefore , that German political parties have emphasized the need for Merkel to raise such matters in her talks with Chinese leaders . However , Merkel\u2019s focus may well remain on trade , investment , and currency flows . After all , unless she can turn ar
 ound the eurozone , her rising profile at home and in Europe could easily wither . What this means is that to secure German leadership of Europe , and her own leadership of Germany , Merkel has to \u201c walk on two legs , \u201d so to speak . She needs to balance both geo-economic and geopolitical factors \u2013 both interests and values \u2013 in advancing Germany\u2019s relations with China . </text>
+<text id="syndicate/ms--europe-or-frau-germania-">
+Ms. Europe or Frau Germania ? BERLIN \u2013 Just what is the matter with Angela Merkel ? Only a short while ago , she was celebrated as \u201c Ms. Europe \u201d ; now , she increasingly gives the impression of being Frau Germania . Instead of providing resolute leadership in the global financial and economic crisis , the European Union\u2019s largest economy is withdrawing into its shell . Germany has always been the motor of European integration , in accordance with its political and economic interests . Every post-World War II government has always been willing to harness Germany\u2019s financial strength to further that aim , i. e. , to foot the bill for Europe . The motto was simple : Germany gives and profits in turn . Should Germany spurn the first part of this formula , the European project would suffer serious damage \u2013 and so will German national interests . Yet this is the direction in which Chancellor Merkel seems to be heading . Indeed , in the wake of the Greek crisis , Merkel spoke 
 publicly of the need to be able to exclude eurozone members that do not play by the rules . Does she seriously believe that the euro and the EU would survive such punitive action ?

<TRUNCATED>