tarrien/Mastodon
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Use charlock_holmes instead of nkf at FetchLinkCardService (#4080)

Browse Source 
* Specs for language detection

* Use CharlockHolmes instead of NKF

* Correct mistakes

* Correct style

* Set hint_enc instead of falling back and strip_tags

* Improve specs

* Add dependencies
This commit is contained in:
nullkal
2017-07-09 05:44:31 +09:00
committed by Eugen Rochko
parent 794781d121
commit 007ab330e6
11 changed files with 78 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
spec/fixtures/requests/sjis.txt vendored
View File

@ -11,10 +11,10 @@ Accept-Ranges: bytes
<HEAD>
<META NAME="GENERATOR" CONTENT="Adobe PageMill 3.0J Mac">
<META HTTP-EQUIV="Content-Type" CONTENT="text/html;CHARSET=x-sjis">
<TITLE>JSIS<EFBFBD>̃y<EFBFBD>[<5B>W</TITLE>
<TITLE>SJIS<EFBFBD>̃y<EFBFBD>[<5B>W</TITLE>
</HEAD>
<BODY>
<P><CENTER><B><FONT SIZE="+2">SJIS<EFBFBD>̃y<EFBFBD>[<5B>W</FONT></B><BR>
<P><CENTER><B><FONT SIZE="+2"><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>N<EFBFBD>܂<EFBFBD><EFBFBD>Ă<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>L<EFBFBD>O<EFBFBD>l<EFBFBD><EFBFBD><EFBFBD>Ă<EFBFBD><EFBFBD>̂̎<EFBFBD><EFBFBD>ł<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>ł<EFBFBD><EFBFBD>B<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>ԂɈӖ<EFBFBD><EFBFBD>҂͐<EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>ǂ<EFBFBD><EFBFBD>Ȕ<EFBFBD><EFBFBD><EFBFBD><EFBFBD>܂<EFBFBD><EFBFBD><EFBFBD><EFBFBD>܂ł<EFBFBD><EFBFBD>\<5C><><EFBFBD>グ<EFBFBD><E382B0><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>邽<EFBFBD>ɂ͎Q<CD8E>l<EFBFBD>A<EFBFBD>邽<EFBFBD><E982BD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>A<EFBFBD><41><EFBFBD><EFBFBD><EFBFBD>ɂ<EFBFBD><C982><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>܂<EFBFBD><DC82>Ȃ<EFBFBD><C882>B<EFBFBD><42><EFBFBD><EFBFBD><EFBFBD>炢<EFBFBD><E782A2><EFBFBD>Ȃ<EFBFBD><C882>̂͂ǂ<CD82><C782><EFBFBD><EFBFBD>㌎<EFBFBD><E38C8E><EFBFBD>ł<EFBFBD><C582>邾<EFBFBD><E982BE><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>B<EFBFBD><42><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>ĉ<EFBFBD><C489>c<EFBFBD><63><EFBFBD><EFBFBD><EFBFBD>ɔ<EFBFBD><C994>R<EFBFBD>K<EFBFBD><4B><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>ɉ]<5D><><EFBFBD>ł<EFBFBD><C582><EFBFBD><EFBFBD><EFBFBD><EFBFBD>͂<EFBFBD><CD82><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>͂<EFBFBD><CD82>Ȃ<EFBFBD><C882><EFBFBD><EFBFBD>w<EFBFBD>}<7D><><EFBFBD>Ƃ<EFBFBD><C682><EFBFBD><EFBFBD><EFBFBD><EFBFBD>o<EFBFBD><6F><EFBFBD><EFBFBD><EFBFBD>Ȃ<EFBFBD><C882><EFBFBD><EFBFBD><EFBFBD><EFBFBD>Ȃ<EFBFBD><C882><EFBFBD><EFBFBD>āA<C481><41><EFBFBD>͎̐̂<CC82><CD8E><EFBFBD><EFBFBD><EFBFBD><EFBFBD>͉A<CD89><41><EFBFBD>{<7B>炩<EFBFBD><E782A9><EFBFBD>A<EFBFBD>v<EFBFBD><76><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>̂<EFBFBD><CC82>̂<EFBFBD><CC82><EFBFBD><EFBFBD><EFBFBD><EFBFBD>̂<EFBFBD><CC82>‚<EFBFBD><C282><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>ɂ<EFBFBD><C982><EFBFBD><EFBFBD>]<5D>ƌ<EFBFBD><C68C><EFBFBD><EFBFBD>΂<EFBFBD><CE82><EFBFBD>man<61>ɂ<EFBFBD><C982><EFBFBD><EFBFBD><EFBFBD><EFBFBD>֎Q<D68E><51><EFBFBD>悤<EFBFBD>ɓ<EFBFBD><C993><EFBFBD><EFBFBD>ɂ<EFBFBD><C982><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>łȂ<C582><C882>̂ŁA<C581><41><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>\<5C><><EFBFBD>ɕς<C995><CF82><EFBFBD><EFBFBD>Ă<EFBFBD><C482><EFBFBD><EFBFBD>ł<EFBFBD><C582><EFBFBD><EFBFBD>ōl<C58D><6C><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>B<EFBFBD><42><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>Ⴆ<EFBFBD>΂<EFBFBD><CE82><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>Ƃǂ܂<C782><DC82><EFBFBD><EFBFBD>̂<EFBFBD><CC82><EFBFBD><EFBFBD>ۂނ<DB82><DE82>݂Ƃ<DD82><C682><EFBFBD><EFBFBD>ł<EFBFBD><C582>āA<C481><41><EFBFBD>̎<EFBFBD><CC8E><EFBFBD><EFBFBD>ł͐\<5C><><EFBFBD><EFBFBD><EFBFBD>ĂƂ<C482><C682>Đ<EFBFBD><C490>Ԃɕ<D482><C995>ׂ̂ɍs<C98D><73><EFBFBD>Ȃ<EFBFBD><C882><EFBFBD><EFBFBD>ȁB</FONT></B><BR>
<HR><BR>
</BODY>
</HTML>