You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@allura.apache.org by br...@apache.org on 2022/02/09 16:20:22 UTC
[allura] branch db/8408 created (now cce6a78)
This is an automated email from the ASF dual-hosted git repository.
brondsem pushed a change to branch db/8408
in repository https://gitbox.apache.org/repos/asf/allura.git.
at cce6a78 [#8408] raise markdown limits now
This branch includes the following new commits:
new a865a55 [#8408] stronger markdown test. and don't use deprecated unittest methods
new 34ab3ba [#8408] upgrade markdown-checklist extension
new e396a66 [#8408] upgrade pyflakes (not really related)
new 92a3107 [#8408] remove old unused script that involves markdown
new 50b5270 [#8408] start Markdown upgrade, simple cleanups
new c6c2969 [#8408] all markdown extensions need to register themselves
new 70b4885 [#8408] tricky parts of Markdown upgrade
new 5a2acdb [#8408] upgrade to markdown 3.1
new a11fba7 [#8408] upgrade to markdown 3.2; drop attr_list extension
new 301fa0b [#8408] upgrade to markdown 3.3
new cce6a78 [#8408] raise markdown limits now
The 11 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[allura] 02/11: [#8408] upgrade markdown-checklist extension
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brondsem pushed a commit to branch db/8408
in repository https://gitbox.apache.org/repos/asf/allura.git
commit 34ab3ba6ac8f5434484c10fc2e0aef3ecc653e8e
Author: Dave Brondsema <db...@slashdotmedia.com>
AuthorDate: Fri Jan 14 12:35:10 2022 -0500
[#8408] upgrade markdown-checklist extension
---
requirements.txt | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/requirements.txt b/requirements.txt
index 3bc65e7..3cab912 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -78,7 +78,7 @@ markdown==2.6.11
# -r requirements.in
# markdown-checklist
# pypeline
-markdown-checklist==0.4.1
+markdown-checklist==0.4.3
# via -r requirements.in
markupsafe==2.0.1
# via
[allura] 10/11: [#8408] upgrade to markdown 3.3
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brondsem pushed a commit to branch db/8408
in repository https://gitbox.apache.org/repos/asf/allura.git
commit 301fa0bd1b100590723c4070c32c22dff6fdb27d
Author: Dave Brondsema <db...@slashdotmedia.com>
AuthorDate: Tue Jan 18 17:11:03 2022 -0500
[#8408] upgrade to markdown 3.3
---
Allura/allura/tests/test_globals.py | 4 +---
Allura/allura/tests/test_helpers.py | 2 +-
requirements.txt | 2 +-
3 files changed, 3 insertions(+), 5 deletions(-)
diff --git a/Allura/allura/tests/test_globals.py b/Allura/allura/tests/test_globals.py
index 131053b..65a5bd9 100644
--- a/Allura/allura/tests/test_globals.py
+++ b/Allura/allura/tests/test_globals.py
@@ -177,7 +177,7 @@ def test_macro_gittip_button():
assert_equal(
r,
'<div class="markdown_content"><p><iframe height="22pt" src="https://www.gittip.com/test/widget.html" '
- 'style="border: 0; margin: 0; padding: 0;" width="48pt"></iframe></p>\n</div>')
+ 'style="border: 0; margin: 0; padding: 0;" width="48pt"></iframe></p></div>')
def test_macro_neighborhood_feeds():
@@ -486,7 +486,6 @@ Some text in a regular paragraph
'<div class="codehilite"><pre><span></span><code><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span>\n'
' <span class="nb">print</span> <span class="n">i</span>\n'
'</code></pre></div>\n'
- '\n'
'</div>'
)
assert_equal(
@@ -501,7 +500,6 @@ def foo(): pass
~~~~'''),
'<div class="markdown_content"><div class="codehilite"><pre><span></span><code>def foo(): pass\n'
'</code></pre></div>\n'
- '\n'
'</div>'
)
diff --git a/Allura/allura/tests/test_helpers.py b/Allura/allura/tests/test_helpers.py
index 5250db8..71f33cb 100644
--- a/Allura/allura/tests/test_helpers.py
+++ b/Allura/allura/tests/test_helpers.py
@@ -275,7 +275,7 @@ def test_render_any_markup_formatting():
'<div class="markdown_content"><h3 id="foo">foo</h3>\n'
'<div class="codehilite"><pre><span></span><code><span class="nt">'
'<script></span>alert(1)<span class="nt">'
- '</script></span> bar\n</code></pre></div>\n\n</div>')
+ '</script></span> bar\n</code></pre></div>\n</div>')
def test_render_any_markdown_encoding():
diff --git a/requirements.txt b/requirements.txt
index a2d4c66..8c41d65 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -75,7 +75,7 @@ iso8601==0.1.16
# via colander
jinja2==3.0.2
# via -r requirements.in
-markdown==3.2.2
+markdown==3.3.6
# via
# -r requirements.in
# markdown-checklist
[allura] 04/11: [#8408] remove old unused script that involves markdown
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brondsem pushed a commit to branch db/8408
in repository https://gitbox.apache.org/repos/asf/allura.git
commit 92a3107d24b898b8127d87f08494144f320f00ad
Author: Dave Brondsema <db...@slashdotmedia.com>
AuthorDate: Fri Jan 14 12:35:29 2022 -0500
[#8408] remove old unused script that involves markdown
---
scripts/perf/md_perf.py | 134 ------------------------------------------------
1 file changed, 134 deletions(-)
diff --git a/scripts/perf/md_perf.py b/scripts/perf/md_perf.py
deleted file mode 100644
index e12f088..0000000
--- a/scripts/perf/md_perf.py
+++ /dev/null
@@ -1,134 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-"""
-A script for timing/profiling the Markdown conversion of an artifact discussion thread.
-
-Example usage:
-
-(env-allura)root@h1v1024:/var/local/allura/Allura(master)$ time paster script production.ini allura/scripts/md_perf.py -- --converter=forge
- Conversion Time (s) Text Size Post._id
- 1 0.021863937377929688 20 7d3ce86bb2c3afaee21f28547ed542d0c8c483b8
- 2 0.002238035202026367 12 96b1283c6db619cd66ee092551758dc1ad5b958f
- 3 0.001832962036132812 9 42c44cd72815d29069378353745bef953bdb2b98
- 4 0.011796951293945312 901 cadbea52291c58bb190597c1df94527369b405fa
- 5 0.143218040466308594 11018 362bee7ef1ba4e062712bda22cb96ae9fe459b95
- 6 0.029596090316772461 1481 46b78785b849f55d24cf32ee1f4e4c916cb7a6fe
- 7 0.045151948928833008 4144 4e2f62188080baba9b88c045ba1f7c26903bfcf9
- 8 0.637056827545166016 14553 51c87f2b1d55c08147bb2353e20361236232b432
- 9 4.510329008102416992 23016 f298ad88584f3dc9b5c37ffde61391bf03d5bae6
- 10 3.614714860916137695 23016 8775a230b19c231daa1608bb4cea1e17aab54550
- 11 0.393079042434692383 19386 ed79e0d2f89366043a007e0837e98945453508c9
- 12 0.000739097595214844 43659 56ac2437ec1bd886a03d23e45aa1c947729760ec
- 13 0.258469104766845703 8520 57152cefe0424b7fc9dff7ab4d21a6ef6e90bf82
- 14 0.264052867889404297 8518 6788b13c90c9719ba46196a0a773f4c228df9f2a
- 15 0.995774984359741211 14553 3f5cce06d2400bcd56d7b90d12c569935f0099a4
-Total time: 10.930670023
-
-real 0m14.256s
-user 0m12.749s
-sys 0m1.112s
-
-"""
-
-from __future__ import unicode_literals
-from __future__ import print_function
-from __future__ import absolute_import
-import argparse
-import cProfile
-import time
-
-try:
- import re2
- re2.set_fallback_notification(re2.FALLBACK_WARNING)
- RE2_INSTALLED = True
-except ImportError:
- RE2_INSTALLED = False
-
-from tg import app_globals as g
-
-MAX_OUTPUT = 99999
-DUMMYTEXT = None
-
-
-def get_artifact():
- from forgeblog import model as BM
- return BM.BlogPost.query.get(
- slug='2013/09/watch-breaking-bad-season-5-episode-16-felina-live-streaming')
-
-
-def main(opts):
- import markdown
- if opts.re2 and RE2_INSTALLED:
- markdown.inlinepatterns.re = re2
- converters = {
- 'markdown': lambda: markdown.Markdown(),
- 'markdown_safe': lambda: markdown.Markdown(safe_mode=True),
- 'markdown_escape': lambda: markdown.Markdown(safe_mode='escape'),
- 'forge': lambda: g.markdown,
- }
- md = converters[opts.converter]()
- artifact = get_artifact()
- return render(artifact, md, opts)
-
-
-def render(artifact, md, opts):
- start = begin = time.time()
- print("%4s %20s %10s %s" % ('', 'Conversion Time (s)', 'Text Size', 'Post._id'))
- for i, p in enumerate(artifact.discussion_thread.posts):
- text = DUMMYTEXT or p.text
- if opts.n and i + 1 not in opts.n:
- print('Skipping post %s' % str(i + 1))
- continue
- if opts.profile:
- print('Profiling post %s' % str(i + 1))
- cProfile.runctx('output = md.convert(text)', globals(), locals())
- else:
- output = md.convert(text)
- elapsed = time.time() - start
- print("%4s %1.18f %10s %s" % (i + 1, elapsed, len(text), p._id))
- if opts.output:
- print('Input:', text[:min(300, len(text))])
- print('Output:', output[:min(MAX_OUTPUT, len(output))])
- start = time.time()
- print("Total time:", start - begin)
- return output
-
-
-def parse_options():
- parser = argparse.ArgumentParser()
- parser.add_argument('--converter', default='markdown')
- parser.add_argument('--profile', action='store_true',
- help='Run profiler and output timings')
- parser.add_argument('--output', action='store_true',
- help='Print result of markdown conversion')
- parser.add_argument('--re2', action='store_true',
- help='Run with re2 instead of re')
- parser.add_argument('--compare', action='store_true',
- help='Run with re and re2, and compare results')
- parser.add_argument('-n', '--n', nargs='+', type=int,
- help='Only convert nth post(s) in thread')
- return parser.parse_args()
-
-
-if __name__ == '__main__':
- opts = parse_options()
- out1 = main(opts)
- if opts.compare:
- opts.re2 = not opts.re2
- out2 = main(opts)
- print('re/re2 outputs match: ', out1 == out2)
[allura] 05/11: [#8408] start Markdown upgrade, simple cleanups
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brondsem pushed a commit to branch db/8408
in repository https://gitbox.apache.org/repos/asf/allura.git
commit 50b527080b90d2fd50be38abe7ecb3067bd972b4
Author: Dave Brondsema <db...@slashdotmedia.com>
AuthorDate: Fri Jan 14 13:21:30 2022 -0500
[#8408] start Markdown upgrade, simple cleanups
---
Allura/allura/lib/app_globals.py | 6 +++---
Allura/allura/lib/markdown_extensions.py | 22 +++++-----------------
Allura/allura/lib/search.py | 2 +-
Allura/allura/tests/test_markdown.py | 2 +-
requirements.in | 2 +-
requirements.txt | 2 +-
6 files changed, 12 insertions(+), 24 deletions(-)
diff --git a/Allura/allura/lib/app_globals.py b/Allura/allura/lib/app_globals.py
index 84c32fa..e6f703a 100644
--- a/Allura/allura/lib/app_globals.py
+++ b/Allura/allura/lib/app_globals.py
@@ -85,7 +85,7 @@ class ForgeMarkdown(markdown.Markdown):
escaped = cgi.escape(h.really_unicode(source))
return Markup('<pre>%s</pre>' % escaped)
try:
- return markdown.Markdown.convert(self, source)
+ return super().convert(source)
except Exception:
log.info('Invalid markdown: %s Upwards trace is %s', source,
''.join(traceback.format_stack()), exc_info=True)
@@ -451,7 +451,7 @@ class Globals(object):
ForgeExtension(**kwargs), EmojiExtension(), UserMentionExtension(),
'markdown.extensions.tables', 'markdown.extensions.toc', 'markdown.extensions.nl2br',
'markdown_checklist.extension'],
- output_format='html4')
+ output_format='html')
@property
def markdown(self):
@@ -473,7 +473,7 @@ class Globals(object):
"""
app = getattr(c, 'app', None)
return ForgeMarkdown(extensions=[CommitMessageExtension(app), EmojiExtension(), 'markdown.extensions.nl2br'],
- output_format='html4')
+ output_format='html')
@property
def production_mode(self):
diff --git a/Allura/allura/lib/markdown_extensions.py b/Allura/allura/lib/markdown_extensions.py
index ba36796..074ac48 100644
--- a/Allura/allura/lib/markdown_extensions.py
+++ b/Allura/allura/lib/markdown_extensions.py
@@ -247,7 +247,7 @@ class ForgeExtension(markdown.Extension):
self._is_email = email
self._macro_context = macro_context
- def extendMarkdown(self, md, md_globals):
+ def extendMarkdown(self, md):
md.registerExtension(self)
md.preprocessors.add('macro_include', ForgeMacroIncludePreprocessor(md), '_end')
# this has to be before the 'escape' processor, otherwise weird
@@ -277,20 +277,14 @@ class ForgeExtension(markdown.Extension):
class EmojiExtension(markdown.Extension):
- EMOJI_RE = '(%s[a-zA-Z0-9\+\-_&.ô’Åéãíç()!#*]+%s)' % (':', ':')
-
- def __init__(self, **kwargs):
- markdown.Extension.__init__(self)
+ EMOJI_RE = r'(:[a-zA-Z0-9\+\-_&.ô’Åéãíç()!#\*]+:)'
- def extendMarkdown(self, md, md_globals):
+ def extendMarkdown(self, md):
md.inlinePatterns["emoji"] = EmojiInlinePattern(self.EMOJI_RE)
class EmojiInlinePattern(markdown.inlinepatterns.Pattern):
-
- def __init__(self, pattern):
- markdown.inlinepatterns.Pattern.__init__(self, pattern)
-
+
def handleMatch(self, m):
emoji_code = m.group(2)
return emoji.emojize(emoji_code, use_aliases=True)
@@ -300,18 +294,12 @@ class UserMentionExtension(markdown.Extension):
UM_RE = r'\B(@(?![0-9]+$)(?!-)[a-z0-9_-]{2,14}[a-z0-9_])'
- def __init__(self, **kwargs):
- markdown.Extension.__init__(self)
-
- def extendMarkdown(self, md, md_globals):
+ def extendMarkdown(self, md):
md.inlinePatterns["user_mentions"] = UserMentionInlinePattern(self.UM_RE)
class UserMentionInlinePattern(markdown.inlinepatterns.Pattern):
- def __init__(self, pattern):
- markdown.inlinepatterns.Pattern.__init__(self, pattern)
-
def handleMatch(self, m):
user_name = m.group(2).replace("@", "")
user = M.User.by_username(user_name)
diff --git a/Allura/allura/lib/search.py b/Allura/allura/lib/search.py
index 6d820cf..a8597f2 100644
--- a/Allura/allura/lib/search.py
+++ b/Allura/allura/lib/search.py
@@ -375,7 +375,7 @@ def find_shortlinks(text):
md = markdown.Markdown(
extensions=['markdown.extensions.codehilite', ForgeExtension(), 'markdown.extensions.tables'],
- output_format='html4')
+ output_format='html')
md.convert(text)
link_index = md.treeprocessors['links'].alinks
return [link for link in link_index if link is not None]
diff --git a/Allura/allura/tests/test_markdown.py b/Allura/allura/tests/test_markdown.py
index a276b67..3a7f3c4 100644
--- a/Allura/allura/tests/test_markdown.py
+++ b/Allura/allura/tests/test_markdown.py
@@ -146,5 +146,5 @@ Not *strong* or _underlined_."""
md = ForgeMarkdown(
extensions=[mde.CommitMessageExtension(app), 'markdown.extensions.nl2br'],
- output_format='html4')
+ output_format='html')
self.assertEqual(md.convert(text), expected_html)
diff --git a/requirements.in b/requirements.in
index 0226d27..f62f95a 100644
--- a/requirements.in
+++ b/requirements.in
@@ -15,7 +15,7 @@ FormEncode
GitPython<=3.1.23
html5lib
Jinja2
-Markdown<3
+Markdown
markdown-checklist
MarkupSafe
Ming
diff --git a/requirements.txt b/requirements.txt
index 287fc9f..8f0ec4f 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -73,7 +73,7 @@ iso8601==0.1.16
# via colander
jinja2==3.0.2
# via -r requirements.in
-markdown==2.6.11
+markdown==3.0.1
# via
# -r requirements.in
# markdown-checklist
[allura] 01/11: [#8408] stronger markdown test. and don't use deprecated unittest methods
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brondsem pushed a commit to branch db/8408
in repository https://gitbox.apache.org/repos/asf/allura.git
commit a865a557e33e3bb0b84628c15626abf0891a16af
Author: Dave Brondsema <db...@slashdotmedia.com>
AuthorDate: Tue Jan 18 12:09:49 2022 -0500
[#8408] stronger markdown test. and don't use deprecated unittest methods
---
Allura/allura/tests/test_globals.py | 54 ++++++++++++++++++++++++++++++-------
AlluraTest/alluratest/tools.py | 4 +--
2 files changed, 46 insertions(+), 12 deletions(-)
diff --git a/Allura/allura/tests/test_globals.py b/Allura/allura/tests/test_globals.py
index 8a38cc7..3a3ecd0 100644
--- a/Allura/allura/tests/test_globals.py
+++ b/Allura/allura/tests/test_globals.py
@@ -449,27 +449,61 @@ def test_markdown_big_text():
def test_markdown_basics():
with h.push_context('test', 'wiki', neighborhood='Projects'):
text = g.markdown.convert('# Foo!\n[Home]')
- assert '<a class="alink" href="/p/test/wiki/Home/">[Home]</a>' in text, text
+ assert_equal(text,
+ '<div class="markdown_content"><h1 id="foo">Foo!</h1>\n'
+ '<p><a class="alink" href="/p/test/wiki/Home/">[Home]</a></p></div>')
text = g.markdown.convert('# Foo!\n[Rooted]')
- assert '<a href=' not in text, text
+ assert_equal(text,
+ '<div class="markdown_content"><h1 id="foo">Foo!</h1>\n'
+ '<p><span>[Rooted]</span></p></div>')
- assert '<br' in g.markdown.convert('Multi\nLine'), g.markdown.convert('Multi\nLine')
- assert '<br' not in g.markdown.convert('Multi\n\nLine')
+ assert_equal(
+ g.markdown.convert('Multi\nLine'),
+ '<div class="markdown_content"><p>Multi<br/>\n'
+ 'Line</p></div>')
+ assert_equal(
+ g.markdown.convert('Multi\n\nLine'),
+ '<div class="markdown_content"><p>Multi</p>\n'
+ '<p>Line</p></div>')
- g.markdown.convert("<class 'foo'>") # should not raise an exception
- assert '<br' not in g.markdown.convert('''# Header
+ # should not raise an exception:
+ assert_equal(
+ g.markdown.convert("<class 'foo'>"),
+ '''<div class="markdown_content"><p><class 'foo'=""></class></p></div>''')
+
+ assert_equal(
+ g.markdown.convert('''# Header
Some text in a regular paragraph
:::python
for i in range(10):
print i
-''')
- assert_in('http://localhost/', g.forge_markdown(email=True).convert('[Home]'))
- assert 'class="codehilite"' in g.markdown.convert('''
+'''),
+ # no <br
+ '<div class="markdown_content"><h1 id="header">Header</h1>\n'
+ '<p>Some text in a regular paragraph</p>\n'
+ '<div class="codehilite"><pre><span></span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span>\n'
+ ' <span class="nb">print</span> <span class="n">i</span>\n'
+ '</pre></div>\n'
+ '\n'
+ '</div>'
+ )
+ assert_equal(
+ g.forge_markdown(email=True).convert('[Home]'),
+ # uses localhost:
+ '<div class="markdown_content"><p><a class="alink" href="http://localhost/p/test/wiki/Home/">[Home]</a></p></div>'
+ )
+ assert_equal(
+ g.markdown.convert('''
~~~~
def foo(): pass
-~~~~''')
+~~~~'''),
+ '<div class="markdown_content"><div class="codehilite"><pre><span></span>def foo(): pass\n'
+ '</pre></div>\n'
+ '\n'
+ '</div>'
+ )
def test_markdown_autolink():
diff --git a/AlluraTest/alluratest/tools.py b/AlluraTest/alluratest/tools.py
index 71a9a6a..efd93e9 100644
--- a/AlluraTest/alluratest/tools.py
+++ b/AlluraTest/alluratest/tools.py
@@ -50,7 +50,7 @@ def assert_equal(*a, **kw):
def assert_equals(*a, **kw):
- return testcase.assertEquals(*a, **kw)
+ return testcase.assertEqual(*a, **kw)
def assert_not_equal(*a, **kw):
@@ -102,7 +102,7 @@ def assert_greater_equal(*a, **kw):
def assert_regexp_matches(*a, **kw):
- return testcase.assertRegexpMatches(*a, **kw)
+ return testcase.assertRegex(*a, **kw)
#
[allura] 07/11: [#8408] tricky parts of Markdown upgrade
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brondsem pushed a commit to branch db/8408
in repository https://gitbox.apache.org/repos/asf/allura.git
commit 70b4885452ef34c7d400c31886e66ba77f8a38a8
Author: Dave Brondsema <db...@slashdotmedia.com>
AuthorDate: Mon Jan 17 12:47:16 2022 -0500
[#8408] tricky parts of Markdown upgrade
---
Allura/allura/lib/markdown_extensions.py | 58 ++++++++++++++++++++++----------
Allura/allura/tests/test_helpers.py | 5 +--
2 files changed, 44 insertions(+), 19 deletions(-)
diff --git a/Allura/allura/lib/markdown_extensions.py b/Allura/allura/lib/markdown_extensions.py
index cfef8e7..826a3ae 100644
--- a/Allura/allura/lib/markdown_extensions.py
+++ b/Allura/allura/lib/markdown_extensions.py
@@ -21,6 +21,7 @@ from __future__ import unicode_literals
from __future__ import absolute_import
import re
import logging
+from typing import List
from six.moves.urllib.parse import urljoin
@@ -44,6 +45,32 @@ log = logging.getLogger(__name__)
MACRO_PATTERN = r'\[\[([^\]\[]+)\]\]'
+# SHORT_REF_RE copied from markdown pre 3.0
+SHORT_REF_RE = markdown.inlinepatterns.NOIMG + r'\[([^\]]+)\]'
+
+# FORGE_LINK_RE copied from markdown pre 3.0's LINK_RE
+NOBRACKET = r'[^\]\[]*'
+BRK = (
+ r'\[(' +
+ (NOBRACKET + r'(\[')*6 +
+ (NOBRACKET + r'\])*')*6 +
+ NOBRACKET + r')\]'
+)
+FORGE_LINK_RE = markdown.inlinepatterns.NOIMG + BRK + \
+ r'''\(\s*(<.*?>|((?:(?:\(.*?\))|[^\(\)]))*?)\s*((['"])(.*?)\12\s*)?\)'''
+
+
+def clear_markdown_registry(reg: markdown.util.Registry, keep: List[str] = []):
+ keep_items = {}
+ for name in keep:
+ keep_items[name] = reg[name]
+
+ # this resets Registry's internal data structures to be empty
+ reg.__init__()
+
+ for name, item in keep_items.items():
+ reg.register(item, name, 50) # arbitrary priority :(
+
class CommitMessageExtension(markdown.Extension):
@@ -70,22 +97,20 @@ class CommitMessageExtension(markdown.Extension):
"""
def __init__(self, app):
- markdown.Extension.__init__(self)
+ super().__init__()
self.app = app
self._use_wiki = False
- def extendMarkdown(self, md, md_globals):
+ def extendMarkdown(self, md):
md.registerExtension(self)
# remove default preprocessors and add our own
- md.preprocessors.clear()
+ clear_markdown_registry(md.preprocessors)
md.preprocessors['trac_refs'] = PatternReplacingProcessor(TracRef1(), TracRef2(), TracRef3(self.app))
# remove all inlinepattern processors except short refs and links
- md.inlinePatterns.clear()
- md.inlinePatterns["link"] = markdown.inlinepatterns.LinkPattern(markdown.inlinepatterns.LINK_RE, md)
- md.inlinePatterns['short_reference'] = ForgeLinkPattern(markdown.inlinepatterns.SHORT_REF_RE, md, ext=self)
+ clear_markdown_registry(md.inlinePatterns, keep=['link'])
+ md.inlinePatterns['short_reference'] = ForgeLinkPattern(SHORT_REF_RE, md, ext=self)
# remove all default block processors except for paragraph
- md.parser.blockprocessors.clear()
- md.parser.blockprocessors['paragraph'] = markdown.blockprocessors.ParagraphProcessor(md.parser)
+ clear_markdown_registry(md.parser.blockprocessors, keep=['paragraph'])
# wrap artifact link text in square brackets
self.forge_link_tree_processor = ForgeLinkTreeProcessor(md)
md.treeprocessors['links'] = self.forge_link_tree_processor
@@ -242,7 +267,7 @@ class PatternReplacingProcessor(markdown.preprocessors.Preprocessor):
class ForgeExtension(markdown.Extension):
def __init__(self, wiki=False, email=False, macro_context=None):
- markdown.Extension.__init__(self)
+ super().__init__()
self._use_wiki = wiki
self._is_email = email
self._macro_context = macro_context
@@ -257,8 +282,8 @@ class ForgeExtension(markdown.Extension):
AutolinkPattern(r'(http(?:s?)://[a-zA-Z0-9./\-\\_%?&=+#;~:!]+)', md),
'<escape')
# replace the link pattern with our extended version
- md.inlinePatterns['link'] = ForgeLinkPattern(markdown.inlinepatterns.LINK_RE, md, ext=self)
- md.inlinePatterns['short_reference'] = ForgeLinkPattern(markdown.inlinepatterns.SHORT_REF_RE, md, ext=self)
+ md.inlinePatterns['link'] = ForgeLinkPattern(FORGE_LINK_RE, md, ext=self)
+ md.inlinePatterns['short_reference'] = ForgeLinkPattern(SHORT_REF_RE, md, ext=self)
# macro must be processed before links
md.inlinePatterns.add('macro', ForgeMacroPattern(MACRO_PATTERN, md, ext=self), '<link')
self.forge_link_tree_processor = ForgeLinkTreeProcessor(md)
@@ -317,13 +342,13 @@ class UserMentionInlinePattern(markdown.inlinepatterns.Pattern):
return result
-class ForgeLinkPattern(markdown.inlinepatterns.LinkPattern):
+class ForgeLinkPattern(markdown.inlinepatterns.Pattern):
artifact_re = re.compile(r'((.*?):)?((.*?):)?(.+)')
def __init__(self, *args, **kwargs):
self.ext = kwargs.pop('ext')
- markdown.inlinepatterns.LinkPattern.__init__(self, *args, **kwargs)
+ super().__init__(*args, **kwargs)
def handleMatch(self, m):
el = markdown.util.etree.Element('a')
@@ -347,7 +372,7 @@ class ForgeLinkPattern(markdown.inlinepatterns.LinkPattern):
return '[TOC]' # skip TOC
if self.artifact_re.match(href):
href, classes = self._expand_alink(href, is_link_with_brackets)
- el.set('href', self.sanitize_url(self.unescape(href.strip())))
+ el.set('href', self.unescape(href.strip()))
el.set('class', classes)
else:
el.set('href', '')
@@ -394,7 +419,7 @@ class ForgeMacroPattern(markdown.inlinepatterns.Pattern):
def __init__(self, *args, **kwargs):
self.ext = kwargs.pop('ext')
self.macro = macro.parse(self.ext._macro_context)
- markdown.inlinepatterns.Pattern.__init__(self, *args, **kwargs)
+ super().__init__(*args, **kwargs)
def handleMatch(self, m):
html = self.macro(m.group(2))
@@ -503,8 +528,7 @@ class HTMLSanitizer(markdown.postprocessors.Postprocessor):
class AutolinkPattern(markdown.inlinepatterns.Pattern):
def __init__(self, pattern, markdown_instance=None):
- markdown.inlinepatterns.Pattern.__init__(
- self, pattern, markdown_instance)
+ super().__init__(pattern, markdown_instance)
# override the complete regex, requiring the preceding text (.*?) to end
# with whitespace or beginning of line "\s|^"
self.compiled_re = re.compile("^(.*?\s|^)%s(.*?)$" % pattern,
diff --git a/Allura/allura/tests/test_helpers.py b/Allura/allura/tests/test_helpers.py
index d743a95..535b525 100644
--- a/Allura/allura/tests/test_helpers.py
+++ b/Allura/allura/tests/test_helpers.py
@@ -270,8 +270,9 @@ def test_render_any_markup_plain():
def test_render_any_markup_formatting():
- assert_equals(h.render_any_markup('README.md', '### foo\n'
- ' <script>alert(1)</script> bar'),
+ # this is broken until markdown 3.1 fixes it again
+ assert_equals(str(h.render_any_markup('README.md', '### foo\n'
+ ' <script>alert(1)</script> bar')),
'<div class="markdown_content"><h3 id="foo">foo</h3>\n'
'<div class="codehilite"><pre><span></span><span class="nt">'
'<script></span>alert(1)<span class="nt">'
[allura] 09/11: [#8408] upgrade to markdown 3.2; drop attr_list extension
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brondsem pushed a commit to branch db/8408
in repository https://gitbox.apache.org/repos/asf/allura.git
commit a11fba73d1c54f09fff6aae551fec01fa39adc71
Author: Dave Brondsema <db...@slashdotmedia.com>
AuthorDate: Tue Jan 18 13:30:16 2022 -0500
[#8408] upgrade to markdown 3.2; drop attr_list extension
attr_list is rarely used, and rarely would we want end-users to set CSS
classes or IDs anyway (although they could through HTML). Most importantly
it is unexpected syntax for users and so curly braces can be interpreted
not how the user wants (and be removed from output text then)
---
Allura/allura/lib/app_globals.py | 3 ++-
Allura/allura/lib/markdown_extensions.py | 9 +++++----
Allura/allura/tests/test_globals.py | 17 ++++++-----------
Allura/allura/tests/test_helpers.py | 4 ++--
ForgeGit/forgegit/tests/functional/test_controllers.py | 4 ++--
requirements.txt | 10 ++++++++--
6 files changed, 25 insertions(+), 22 deletions(-)
diff --git a/Allura/allura/lib/app_globals.py b/Allura/allura/lib/app_globals.py
index e6f703a..44d81f3 100644
--- a/Allura/allura/lib/app_globals.py
+++ b/Allura/allura/lib/app_globals.py
@@ -447,7 +447,8 @@ class Globals(object):
'''return a markdown.Markdown object on which you can call convert'''
return ForgeMarkdown(
extensions=['markdown.extensions.fenced_code', 'markdown.extensions.codehilite',
- 'markdown.extensions.extra', # to allow markdown inside HTML tags
+ 'markdown.extensions.abbr', 'markdown.extensions.def_list', 'markdown.extensions.footnotes',
+ 'markdown.extensions.md_in_html',
ForgeExtension(**kwargs), EmojiExtension(), UserMentionExtension(),
'markdown.extensions.tables', 'markdown.extensions.toc', 'markdown.extensions.nl2br',
'markdown_checklist.extension'],
diff --git a/Allura/allura/lib/markdown_extensions.py b/Allura/allura/lib/markdown_extensions.py
index 826a3ae..27303c0 100644
--- a/Allura/allura/lib/markdown_extensions.py
+++ b/Allura/allura/lib/markdown_extensions.py
@@ -22,6 +22,7 @@ from __future__ import absolute_import
import re
import logging
from typing import List
+import xml.etree.ElementTree as etree
from six.moves.urllib.parse import urljoin
@@ -333,7 +334,7 @@ class UserMentionInlinePattern(markdown.inlinepatterns.Pattern):
result = None
if user and not user.pending and not user.disabled:
- result = markdown.util.etree.Element('a')
+ result = etree.Element('a')
result.text = "@%s" % user_name
result.set('href', user.url())
result.set('class', 'user-mention')
@@ -351,7 +352,7 @@ class ForgeLinkPattern(markdown.inlinepatterns.Pattern):
super().__init__(*args, **kwargs)
def handleMatch(self, m):
- el = markdown.util.etree.Element('a')
+ el = etree.Element('a')
el.text = m.group(2)
is_link_with_brackets = False
try:
@@ -383,7 +384,7 @@ class ForgeLinkPattern(markdown.inlinepatterns.Pattern):
if 'notfound' in classes and not self.ext._use_wiki:
text = el.text
- el = markdown.util.etree.Element('span')
+ el = etree.Element('span')
el.text = '[%s]' % text
return el
@@ -536,7 +537,7 @@ class AutolinkPattern(markdown.inlinepatterns.Pattern):
def handleMatch(self, mo):
old_link = mo.group(2)
- result = markdown.util.etree.Element('a')
+ result = etree.Element('a')
result.text = old_link
# since this is run before the builtin 'escape' processor, we have to
# do our own unescaping
diff --git a/Allura/allura/tests/test_globals.py b/Allura/allura/tests/test_globals.py
index 3a3ecd0..131053b 100644
--- a/Allura/allura/tests/test_globals.py
+++ b/Allura/allura/tests/test_globals.py
@@ -483,9 +483,9 @@ Some text in a regular paragraph
# no <br
'<div class="markdown_content"><h1 id="header">Header</h1>\n'
'<p>Some text in a regular paragraph</p>\n'
- '<div class="codehilite"><pre><span></span><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span>\n'
+ '<div class="codehilite"><pre><span></span><code><span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">10</span><span class="p">):</span>\n'
' <span class="nb">print</span> <span class="n">i</span>\n'
- '</pre></div>\n'
+ '</code></pre></div>\n'
'\n'
'</div>'
)
@@ -499,8 +499,8 @@ Some text in a regular paragraph
~~~~
def foo(): pass
~~~~'''),
- '<div class="markdown_content"><div class="codehilite"><pre><span></span>def foo(): pass\n'
- '</pre></div>\n'
+ '<div class="markdown_content"><div class="codehilite"><pre><span></span><code>def foo(): pass\n'
+ '</code></pre></div>\n'
'\n'
'</div>'
)
@@ -522,7 +522,7 @@ def test_markdown_autolink():
g.markdown.convert('a blahttp://sdf.com z'))
assert_in('literal <code>http://domain.net</code> literal',
g.markdown.convert('literal `http://domain.net` literal'))
- assert_in('<pre><span></span>preformatted http://domain.net\n</pre>',
+ assert_in('<pre><span></span><code>preformatted http://domain.net\n</code></pre>',
g.markdown.convert(' :::text\n'
' preformatted http://domain.net'))
@@ -869,12 +869,7 @@ class TestEmojis(unittest.TestCase):
output = g.markdown.convert('```html\n<p>:camel:</p>\n```')
assert_in(':camel:', output)
output = g.markdown.convert('~~~\n:camel:\n~~~')
- try:
- # older pygments
- assert_in('<span></span><span class="err">:camel:</span>', output)
- except AssertionError:
- # newer pygments
- assert_in('<pre><span></span>:camel:\n</pre>', output)
+ assert_in('<pre><span></span><code>:camel:\n</code></pre>', output)
def test_markdown_commit_with_emojis(self):
output = g.markdown_commit.convert('Thumbs up emoji :+1: wow!')
diff --git a/Allura/allura/tests/test_helpers.py b/Allura/allura/tests/test_helpers.py
index d57b598..5250db8 100644
--- a/Allura/allura/tests/test_helpers.py
+++ b/Allura/allura/tests/test_helpers.py
@@ -273,9 +273,9 @@ def test_render_any_markup_formatting():
assert_equals(str(h.render_any_markup('README.md', '### foo\n'
' <script>alert(1)</script> bar')),
'<div class="markdown_content"><h3 id="foo">foo</h3>\n'
- '<div class="codehilite"><pre><span></span><span class="nt">'
+ '<div class="codehilite"><pre><span></span><code><span class="nt">'
'<script></span>alert(1)<span class="nt">'
- '</script></span> bar\n</pre></div>\n\n</div>')
+ '</script></span> bar\n</code></pre></div>\n\n</div>')
def test_render_any_markdown_encoding():
diff --git a/ForgeGit/forgegit/tests/functional/test_controllers.py b/ForgeGit/forgegit/tests/functional/test_controllers.py
index 3a5a474..81fde01 100644
--- a/ForgeGit/forgegit/tests/functional/test_controllers.py
+++ b/ForgeGit/forgegit/tests/functional/test_controllers.py
@@ -902,12 +902,12 @@ class TestFork(_TestCase):
</li>
</ul>
<p>Diff:</p>
-<div class="codehilite"><pre><span></span><span class="gd">--- old</span>
+<div class="codehilite"><pre><span></span><code><span class="gd">--- old</span>
<span class="gi">+++ new</span>
<span class="gu">@@ -1 +1 @@</span>
<span class="gd">-description</span>
<span class="gi">+changed description</span>
-</pre></div>
+</code></pre></div>
</div>
""".strip())
diff --git a/requirements.txt b/requirements.txt
index d2ed980..a2d4c66 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -67,13 +67,15 @@ httplib2==0.19.0
# via oauth2
idna==3.2
# via requests
+importlib-metadata==4.10.1
+ # via markdown
inflection==0.5.1
# via profanityfilter
iso8601==0.1.16
# via colander
jinja2==3.0.2
# via -r requirements.in
-markdown==3.1.1
+markdown==3.2.2
# via
# -r requirements.in
# markdown-checklist
@@ -194,7 +196,9 @@ translationstring==1.4
turbogears2==2.3.12
# via -r requirements.in
typing-extensions==3.10.0.2
- # via gitpython
+ # via
+ # gitpython
+ # importlib-metadata
urllib3==1.26.6
# via requests
waitress==2.0.0
@@ -218,6 +222,8 @@ werkzeug==2.0.2
# via -r requirements.in
wrapt==1.11.2
# via -r requirements.in
+zipp==3.7.0
+ # via importlib-metadata
# The following packages are considered to be unsafe in a requirements file:
# setuptools
[allura] 11/11: [#8408] raise markdown limits now
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brondsem pushed a commit to branch db/8408
in repository https://gitbox.apache.org/repos/asf/allura.git
commit cce6a78a1fcdc5a322787d7243454f3842bf0c05
Author: Dave Brondsema <db...@slashdotmedia.com>
AuthorDate: Tue Feb 8 16:47:11 2022 -0500
[#8408] raise markdown limits now
---
Allura/allura/lib/app_globals.py | 2 +-
Allura/development.ini | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/Allura/allura/lib/app_globals.py b/Allura/allura/lib/app_globals.py
index 44d81f3..834075b 100644
--- a/Allura/allura/lib/app_globals.py
+++ b/Allura/allura/lib/app_globals.py
@@ -78,7 +78,7 @@ log = logging.getLogger(__name__)
class ForgeMarkdown(markdown.Markdown):
def convert(self, source, render_limit=True):
- if render_limit and len(source) > asint(config.get('markdown_render_max_length', 40000)):
+ if render_limit and len(source) > asint(config.get('markdown_render_max_length', 80000)):
# if text is too big, markdown can take a long time to process it,
# so we return it as a plain text
log.info('Text is too big. Skipping markdown processing')
diff --git a/Allura/development.ini b/Allura/development.ini
index 6f4a084..eb024f8 100644
--- a/Allura/development.ini
+++ b/Allura/development.ini
@@ -589,7 +589,7 @@ site_notification.impressions = 0
; posts. Remove entirely to cache nothing.
markdown_cache_threshold = .1
; markdown text longer than max length will not be converted to html
-markdown_render_max_length = 100000
+markdown_render_max_length = 200000
; Don't add rel=nofollow to these domains when generating links from Markdown content
;nofollow_exempt_domains =
[allura] 06/11: [#8408] all markdown extensions need to register themselves
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brondsem pushed a commit to branch db/8408
in repository https://gitbox.apache.org/repos/asf/allura.git
commit c6c2969177eb190262335c0331c3700ff0194dc8
Author: Dave Brondsema <db...@slashdotmedia.com>
AuthorDate: Fri Jan 14 15:57:12 2022 -0500
[#8408] all markdown extensions need to register themselves
---
Allura/allura/lib/markdown_extensions.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/Allura/allura/lib/markdown_extensions.py b/Allura/allura/lib/markdown_extensions.py
index 074ac48..cfef8e7 100644
--- a/Allura/allura/lib/markdown_extensions.py
+++ b/Allura/allura/lib/markdown_extensions.py
@@ -280,6 +280,7 @@ class EmojiExtension(markdown.Extension):
EMOJI_RE = r'(:[a-zA-Z0-9\+\-_&.ô’Åéãíç()!#\*]+:)'
def extendMarkdown(self, md):
+ md.registerExtension(self)
md.inlinePatterns["emoji"] = EmojiInlinePattern(self.EMOJI_RE)
@@ -295,6 +296,7 @@ class UserMentionExtension(markdown.Extension):
UM_RE = r'\B(@(?![0-9]+$)(?!-)[a-z0-9_-]{2,14}[a-z0-9_])'
def extendMarkdown(self, md):
+ md.registerExtension(self)
md.inlinePatterns["user_mentions"] = UserMentionInlinePattern(self.UM_RE)
[allura] 08/11: [#8408] upgrade to markdown 3.1
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brondsem pushed a commit to branch db/8408
in repository https://gitbox.apache.org/repos/asf/allura.git
commit 5a2acdb830e8810900e379f4d67bce84b9e4e85a
Author: Dave Brondsema <db...@slashdotmedia.com>
AuthorDate: Tue Jan 18 13:11:58 2022 -0500
[#8408] upgrade to markdown 3.1
---
Allura/allura/tests/test_helpers.py | 1 -
requirements.txt | 2 +-
2 files changed, 1 insertion(+), 2 deletions(-)
diff --git a/Allura/allura/tests/test_helpers.py b/Allura/allura/tests/test_helpers.py
index 535b525..d57b598 100644
--- a/Allura/allura/tests/test_helpers.py
+++ b/Allura/allura/tests/test_helpers.py
@@ -270,7 +270,6 @@ def test_render_any_markup_plain():
def test_render_any_markup_formatting():
- # this is broken until markdown 3.1 fixes it again
assert_equals(str(h.render_any_markup('README.md', '### foo\n'
' <script>alert(1)</script> bar')),
'<div class="markdown_content"><h3 id="foo">foo</h3>\n'
diff --git a/requirements.txt b/requirements.txt
index 8f0ec4f..d2ed980 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -73,7 +73,7 @@ iso8601==0.1.16
# via colander
jinja2==3.0.2
# via -r requirements.in
-markdown==3.0.1
+markdown==3.1.1
# via
# -r requirements.in
# markdown-checklist
[allura] 03/11: [#8408] upgrade pyflakes (not really related)
Posted by br...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
brondsem pushed a commit to branch db/8408
in repository https://gitbox.apache.org/repos/asf/allura.git
commit e396a66fa111d65cd595a875262de798b52f1c90
Author: Dave Brondsema <db...@slashdotmedia.com>
AuthorDate: Mon Jan 17 12:47:33 2022 -0500
[#8408] upgrade pyflakes (not really related)
---
requirements.txt | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/requirements.txt b/requirements.txt
index 3cab912..287fc9f 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -118,7 +118,7 @@ profanityfilter==2.0.6
# via -r requirements.in
pycparser==2.20
# via cffi
-pyflakes==2.1.1
+pyflakes==2.4.0
# via -r requirements.in
pygments==2.9.0
# via -r requirements.in